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INTRODUCTION 

During the past few years, the issue of skill standards has arisen as a core element in the U.S. 
public debate about how to ensure that education and training endeavors result in a workforce that meets 
employer needs in. an era of global competition. That interest has been stimulated from at least three 
directions. 

First, a few national trade associations such as the Automotive Service Excellence and Associated 
General Contractors have been working voluntarily for a number of years with state vocational education 
leaders to improve occupational education and training efforts of schools. Those efforts, while positive, 
were limited, and state educators have had to reach out to local employers to expand the effort to establish 
relevant curriculum and measures to assess the competencies students acquired while in school. Today, a 
wide array of efforts are underway in every state to involve industry representatives in constructing 
workplace skill requirements. The U.S. Department of Education's Office of Vocational & Adult Education 
has taken on the crucial responsibility of assisting the states and industries in that effort. 

Second, efforts to improve the quality of training programs - ranging from the classic work-based 
apprenticeship programs to the second chance training programs — led the U.S. Department of Labor to the 
same conclusion reached in the vocational education arena. Skill standards, driven by industry, were 
needed. Without them, it would never be possible to measure the effectiveness of programs and judge the 
value of public and private investments in training. 

Third, the bipartisan Commission on the Skills of the American Workforce gave the nascent 
movement an important boost in its influential 1990 report America's Choice: High Skills or Low Wages! 
Successful international competitors, said the Commission, had national systems and structures that 
involved industries in the development and implementation of education and training for large portions of 
the workforce. Among major industrial nations, only the United States lacks such a system, the 
Commission found. 

With this convergence of forces the federal government launched a collaborative effort between 
the two federal agencies to promote the development of a voluntary national skill standards effort that 
brings to the table all of the stakeholders, including industry associations, unions, and educators from both 
secondary and postsecondary levels. It represents a substantia! effort. 

This report is part of that effort. The Office of Vocational and Adult Education recognized there 
was a critical need to first know what is happening in this country before it would be possible to pull 
together all of the pieces within a coherent national framework. It brings together, for the first time, 
explanatory materials regarding what is currently underway within all of the states as well as within trade 
and professional associations regarding the utilization of skill standards systems. 
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This is a descriptive report. It focuses on what is. It does not address the issues of quality or 
value. While voluminous, it is only a beginning. 

What was found was a great deal of effort currently underway in a wide array of quarters. There 
is a rich base to build upon as the federal government moves forward to develop a national framework for 
the further development of a voluntary system of skill standards that will meet the needs of employers and 
future and current employees. 

However, despite the activity already underway, serious gaps are found in the current practice in 
this country, including: 

► Few skill standards systems include levels that can assist an individual in moving from novice to 
master in his or her preferred occupation. 

► In some of our most important competitive sectors, little or no work has been undertaken to 
develop nationwide skill standards. 

► A crazy quilt pattern of financing the components of the system exists today, raising questions 
about both the cost efficiency and effectiveness of our current system, 

► The infrastructure has not adequately supported the development and upgrading of one of the 
most important components of a high quality skill standards system - the instructors. 

► There is no common agreement about what is included in a definition of an industry or an 
occupational grouping or cluster, leading to confusion across the varied skill standards efforts. 
One of the current goals of the federal government skill standards effort is to benchmark the 

standards against world class competitors. It became increasingly clear as this study progressed that in 
many cases this is already occurring. Indeed, it is obvious that some American trade associations and 
education and training efforts are the worldwide pacesetters. The technical processes used to conduct job 
analysis and assessments in this country are being copied all over the world. In other words, we have the 
technical know-how. We do not have the efficient, effective and understandable overarching framework 
within which the skill standards systems can thrive. In the parlance of the quality management experts, 
we lack a common language and common procedures to promote the continuous improvement of the 
system(s). 

This is a "first ever" undertaking; in order to develop a common framework that crosses the 
boundaries of the public and private sectors and considers multiple variations within both of those sectors 
a set of ten organizational topics were developed to organize the materials. 

1 . A Description of the Individual Program or System(s) This includes an overview of existing 
systems, description of the functional purposes for their development (e.g., individual certification 
and curriculum development), identification of target populations, prerequisites for access to the 
system (e.g., participation in training, work experience), courses of study and levels of 
certification, and management issues such as administrative authority and financing of the system. 

2. The Extent of Coverage by the System(s) - The range of industries and occupations covered by 
existing systems, the overlap between industries and occupations, an estimation of the number of 
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workers included, and how the coverage coincides with educational activity and government 
regulation of the occupation and industry. 

For these two topical areas every attempt has been made to be as comprehensive as possible, 
particularly for those occupations that have not traditionally required a four year college degree. This focus 
was selected by the federal government due to the continuing and mounting concerns articulated by 
business and industry representatives that a serious erosion of qualified entry level workers is permeating 
the U.S. labor force. 

To fully understand the evolution of both industry-driven and education-driven systems, additional 
information is provided, where available, that includes the following topics. 

3. The Historical Development of the System(s) -- This includes the driving forces or incentives, 
what barriers were encountered, and the time factors for achieving current levels of usage and 
coverage. 

4. The Partners' Roles and Responsibilities - The varying roles and reasons for involvement of 
educational systems and institutions, industry associations and employers, accrediting bodies, 
labor organizations, general government, apprenticeship programs and other relevant bodies in 
establishing the skill standards systems. 

5. The Standards Setting Processes - The methods used in setting skill standards, including who 
is involved and the time, resources and staff capabilities needed. 

6. The Occupational Analysis Processes -- How occupations are classified and what approaches 
are used in determining the specific job requirements for a single occupation or common grouping 
of occupations. 

7. The Assessment Processes -- The procedures, instruments, processes and types of anticipated 
outcomes vary and will be described. Specific topics include types of testing and selection 
processes, relationships to instruction, associated costs, methodologies to assure consistency and 
integrity, methodologies used to insure against discrimination and other legal limitations, and 
approaches for maintaining and updating the assessment processes. 

8. The Instructional Approaches - Four major topics are explored: the development of curriculum, 
the delivery of curricula, the selection and training of instructors, and the type of sites used for the 
delivery of instruction. 

9. Maintaining and Updating Skill Standards - The factors that affect the maintenance and 
updating of standards, including the time, resources and staff needed. 

10. How Information is Shared and Used in Skill Standards Systems - How, what and with whom 
information is shared by various partners and external groups. How skill standards are used by 
employers in hiring and promotion practices, by educators/trainers in setting program outcomes 
and performance standards, in curriculum development and instruction, by government in 
establishing regulations and licensure policy, and by accrediting organizations. 
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JOB ANALYSIS 

Job analysis is fundamental to an investigation of industry standards. It provides a "reality check", 
not only on what workers do as work, but also on the quality of their work performance. 

Job analysis is the systematic gathering, documenting, and analyzing of information about 
employees acting to perform the tasks incumbent to their jobs within any kind of a work setting. Analysis 
deals with job content, job requirements as well as the context of the entire work organization. 

Among the purposes for which job analysis information is used are job descriptions, job evaluation 
and classification, performance appraisal, training design, work design and selection/promotion systems. 

Job analysis is critical to employment decisions because it is the foundation upon which 
employment tests - and inferences based upon test scores - are built. Job analysis establishes validity 
under the Uniform Guidelines regarding the Civil Rights Act of 1964. Three types of validity are presented 
as legitimate in the Guidelines : {a) content; (b) criterion-related; and (c) construct. Content validity is the 
degree to which some sample of tasks, questions, samples, or items on an assessment accurately 
represents the defined universe of possible content. Criterion-related validity is the degree to which the 
scores on a test accurately correspond to the performance, knowledge, and skills of those at work. 
Construct validity is the degree to which the scores on a test accurately measure a psychological 
characteristic of interest. 

Job analysis techniques are the formal procedures that researchers follow in order to obtain and 
analyze information about specific jobs. Cornelius (1988) determined that the components of the job 
analysis process fall into four broad categories: sources of job data; methods of obtaining job data; type of 
job descriptor; and purpose of the analysis. 

Since the 1950's, a number of specific techniques have been devised. Six major ones are: 

1. Functional Job Analysis - Trained analysts investigate the interactions among work, workers, 
and work organization through a multi-dimensional information collection process. FJA is used 
extensively throughout industry in managerial, professional and craft situations to classify jobs, 
generate work sample tests and to develop performance standards for work. 

2. Critical Incident Technique - Analysts identify critical incidents that illustrate behaviors that are 
effective or ineffective in accomplishing the aims of the job. Critical incidents are then classified 
into categories of behavior. CIT is typically used as a supplemental data collection technique to 
another method. It is time consuming, requires someone with special training, and produces data 
that is not necessarily representative of the range of tasks performed in the occupation. 

3. Position Analysis Questionnaire - The PAQ questionnaire uses almost 200 elements of job 
activity organized into broad categories. It is a combination of contextual information, content and 
work requirements, and allows derivation of job dimension scores for occupations. The PAQ has 
been used to create compensation programs and perform job evaluations. 
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4. Developing a Curriculum DACUM is a newer technique, originated in Canada and popularized 
by vocational education in the United States. The DACUM process involves role incumbents and 
supervisors for up to several days with a trained facilitator to generate the information. The result 
is a listing of tasks and activities for any particular job. According to Dr. Robert Norton of Ohio 
State University, DACUM operates on three assumptions: (a) expert workers can define their job 
more accurately than anyone else; (b) an effective way to describe a job is to define the tasks 
expert workers perform; and (c) all tasks demand certain knowledge, skills, tools, and attitudes in 
order to be performed correctly. 

5. Job Task Inventory (CODAP and the Extended Search) - CODAP is a computer analyzed job 
task inventory approach. Its most frequent use is to plan and/or adjust training programs so they 
more accurately reflect the actual practice of work. 

6. Observation Direct observation provides depth of analysis, validity and flexibility. However, it 
is time consuming, difficult to analyze, and susceptible to subjectivity. 



Assessment and testing is fundamental to any conception of a national standards program. 
Standards programs in other industrialized nations include assessment as a necessary step to certify the 
skills and knowledge of individual trainees. In addition, the information assessment provides is necessary 
to determine the effectiveness of the training programs as well as the accuracy and usefulness of the 
standards program itself. Finally, testing and assessment is a necessary ingredient to permit the match of 
pools of skilled workers and employer manpower needs within labor markets. 



It has become axiomatic that if you have an assessment, testing and certification program, then 
someone will sue you. Understanding the legal issues associated with assessment is crucial to making 
effective, legally supportable policy choices. 

Perhaps the most important law affecting assessment is the Civil Rights Act of 1964, as amended. 
The key concept emanating from Title VII of that act is known as "adverse impact": when neutral- 
appearing selection criteria have the effect of discriminating against a protected class. The U.S. Supreme 
Court has held that if a test has adverse impact, the test must be shown to be valid to be used. A series 
of guidelines have amerged from case law: 



1 . An employer's test must be based upon a thorough job analysis. 

2. The link between the job analysis and the test content must be established. 

3. The content validity of the test must include results from the job analysis that indicate the 
relative importance of the identified work behaviors. 

4. Tests must provide for adequate agreement among raters of applicants. 

5. Ratings should be based on performance dimensions, not personality traits. 
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Even with these and other guidelines, it is important to note that reviews of court rulings make it 
clear that the courts have not agreed on a uniform set of standards for what assessments are legal and 
which are not. But the evidence to date argues to only use methods and tests whose validity can be 
established through some form of job analysis. 

A second important legal issue related to assessment comes within the National Labor Relations 
Act. In the organized sector, labor unions must be included with an equal voice in the creation of 
standards and assessment systems. 

Third, the Americans with Disabilities Act of 1990 prohibits any standards process that involves 
assessment and certification from discriminating on the basis of disability. The test itself must not assess 
the disability, but rather skills and knowledge appropriate to the job. 

A last area of legislation with implications for national skill standards and certification systems is 
tied to licensing. If government takes the lead in establishing national standards and credentials that limit 
people's entry into or advancement within certain kinds of occupations, then the entire process may be 
subject not only to Civil Rights Act claims but also to "equal protection and due process" claims under the 
5th and 14th Amendments. If xhe scheme if voluntary, due process does not apply. 

Test Development Guidelines 

The technical standards for test development, contained in the American Psychological 
Association's Standards for Educational and Psychological Testing (1985) offer the following guidelines: 

1 . Test content and performance criteria must be based on a thorough job analysis. 

2. One must prepare a specification document in which is set forth all the rules for developing 
the assessment. 

3. Competent persons need to be involved in developing the test items and situations. 

4. Assemble the examination form from the items and situations created in the assessment pool 
and based on the directions provided in the Specification Manual. 

5. Develop criteria measures to differentiate between those that succeed in the assessment and 
those that do not succeed in thu assessment. 

6. Establish procedures for insuring fairness for those being assessed. 

7. Train raters and establish inter-rater reliability. 

8. A test administration manual should be provided that insures a standard operating procedure 
for giving the assessment. 

9. Directions must be provided on analyzing and reporting the information. 

10. Provide explanations of formal procedures for test analysis and review. 

11. Provide for examination security. 

12. Develop and maintain specific information on each type of validity necessary to support 'the 
test. 
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Industry Practices in Assessment 

Business and industry use many types of procedures as assessments in order to make employment 
decisions about candidate selection and promotions. Riley and Warech suggest there are at least 18 
different types of assessments used by American employers, as illustrated in Table 1. 

Table #1 
Types of Assessments or Tests 
Used by American Employers* 



1 . Paper and Pencil Tests 



2. Work Samples 



Assessment Centers 



4. Trainability Tests 



Interviews 



6. Peer Evaluations 



7. Self-Assessments 



8. Bio Data 



9. Grades and Educational Achievement 



10. Reference Checks 



1 1 . Seniority Systems 



12. Personality Tests 



13. Projective Techniques 



14. Individual Assessment by Experts 



'/5. Scored Training and Experience Evaluation 16. Handwriting Analysis 



17. Tests of Physical Ability 



18. Integrity Tests 



•Source: Richard R. Reilly and Michael A. Warech. (1988). The Validity and Fairness of Alternative 
Predictors of Occupational Performance . Berkeley: Center for Behavioral Studies at the University of 
California, 1988. 



As comprehensive as this list is, it ignores two methods frequently used by employers: on-the-job 
assessment of actual practice which is frequently used by apprenticeship programs; and portfolio 
assessment or the intentional inspection of work products against some criteria to determine proficiency. 

Historically, employers have relied on the paper and pencil type tests as their primary means of 
assessment. In general, paper and pencil tests for certification have distinct advantages and 
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disadvantages. Advantages include allowing sponsors/test developers to provide excellent statistics on 
reliability, great ease of administration, and comparable practice for scoring. ??owever, often they do not 
accurately reflect content validity in an occupational setting because they emphasize school learned 
information, deal with verbal type of information and emphasize recall. It is very difficult, although not 
impossible, to assess skills. Such tests are expensive to develop although relatively inexpensive to use. 

In apprenticeships, on-the-job assessment occurs on three levels. First, work hours are recorded 
and reported to the Joint Apprenticeship and Training Committee for each apprentice. Second, the 
apprentice's ability to do that work is recorded because the apprentice must continue to work successfully 
on the trade practice to keep his/her job. Third, the apprentice is required to attend related instructional 
courses. 

While a number of apprentice programs are adding to these three traditional assessments by 
moving toward work sample types of assessment that might be termed "authentic assessment", one 
should not assume that paper and pencil tests are unusable, inappropriate or without validity. They are 
useful, especially in combination with other types of tests. Furthermore, even the "authentic" assessments 
have difficulties in insuring that each observer will rate a behavior the same way. Perhaps an even greater 
concern is the cost of providing work sample performance assessments. 

TERMINOLOGY MATTERS 

During the course of this study it became increasingly obvious that in this area of emerging public 
attention and policy development a great amount of confusion exists simply due to the lack of a common 
language. Common understandings and common frames of reference will be essential as the skill 
standards initiative moves forward. 

The following are examples of terminology issues that surfaced during this study: 

► Standards - among the dimensions of variation in use of this term are time vs. skill, minimum 
qualifications vs. optimal performance. 

► Certification refers to both achievement of minimum qualifications for entry into an occupation 
and to achievement of specialty skills. 

► Performance Tests -- some use the term narrowly to refer to demonstration of ability to complete 
work-simulation tasks while others use it broadly to refer to any type of individual assessment. 

► Job Analysis - A series of terms are often used synonymously: job analysis, work analysis and 
occupational analysis. Sometimes job analysis is used as a subset of a larger clustering under the 
term occupational analysis. Another term, "task analysis" is frequently intermingled, either as a 
synonym for job analysis or as a discrete step in the job analysis process. 

► Skill - While the term basically means the ability to do something, confusion often surrounds its 
usage. For example, referring to a discrete behavior vs. an entire set of performances. 
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► Certification - Generally used to refer to voluntary, non-governmental recognition of individuals. 
Certification may require only minimum competency levels, or may recognize advanced levels of 
accomplishment or proficiency. 

► Accreditation - Applies to programs within institutions, rather tnan to individuals. Generally a 
voluntary form of recognition granted by an agency or association to programs or organizations 
that meet established standards. 

► Skill Standards - Varied definitions of this term are used. One common one from education is 
that skill standards are workplace knowledge, skills and attitudes required for acceptable job 
performance. Many states simply use "skill standards" and "competencies" synonymously. 



America has placed a heavy reliance upon a network of predominantly voluntary organizations to 
promote self-regulation of the education and certification systems. Adherence to the criteria promulgated 
by any of the following organizations' rules and standards are strictly voluntary on the part of any 
individual institution or organization. However, their influence over the accreditation and certification 
systems are strong and their experiences and expertise represent a rich foundation that can contribute to 
the development of a voluntary skill standards system for the country. 

Education and Accreditation Overseers 

The American Council on Education, composed of 1,500 institutions of higher learning is an 
umbrella organization that focuses on all aspects of postsecondary education. It operates through a variety 
of voluntary commissions and centers, its Commission on Educational Credit and Credentials guides the 
operations of the educational community in this country, ensuring the integrity of academic credit and 
credentialing as well as recognition of adult learning. 

The Council collaborates with the Council for Adult and Experiential Learning, an independent 
organization that promotes recognition of prior learning assessment gained outside the classroom. 

The Council on Postsecondary Accreditation 1 serves as an umbrella standards setting organization 
for education institutions voluntarily seeking either institutional recognition or to have specific programs 
recognized by a specialized accrediting body. There are regional and national accrediting bodies recognized 
by COPA and the U.S. Department of Education. 

This is a period of some turmoil within the accreditation community as they attempt to sort out the 
appropriate role of accrediting bodies in terms of measuring institutional effectiveness. Traditional 
accreditation measures have focused on "input" criteria such as adequacy of faculty or facility. There is 



* Note: In the spring of 1993, COPA voted to dissolve itself. As of this writing, key organizations 
involved in accreditation are exploring alternative organized structures. 
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now movement toward requiring institutions to have some form of measuring outcomes such as showing 
how many of their students were able to pass a nationally recognized certification exam. 



Certifiers' Overseers 

The American National Standards Institute (ANSI), formed in 1918, is an umbrella organization that 
approves standards established by technical and professional societies, trade associations and individual 
companies. Over 9,500 standards have been approved by ANSI, mostly for products. ANSI is the U.S. 
member of the International Organization for Standards (ISO). All of the other countries are represented by 
the central gcvernments. ISO standards, while voluntary, are increasingly being used to assure 
conformance of products and services across international boundaries. The ISO 9000 series is a set of 
technical documents that put forth criteria focused on quality assurance. Embedded in these standards are 
criteria related to the quality of the workforce. Companies wanting to participate in the European market 
are increasingly being required to show proof that workers have the appropriate credentials to perform their 
jobs. 

ANSI's criteria for approving any accreditation program include (but are not limited to): 

► Be operated by a certification body with independent third-party status; 

► Have clear procedural guidelines; 

► Provide full disclosure of information about the accreditation activity; and 

► Have an annual review of the program by ANSI. 

A second overseer, the National Commission for Certifying Agencies, was created in 1989. It was 
established by the parent National Organization for Competency Assurance to establish national voluntary 
standards for and recognize compliance with those standards by agencies certifying individuals in a wide 
range of professions and occupations. The parent NOCA is a membership organization whose mission is to 
provide a forum for all types of practitioners and organizations interested in competency assurance and 
certification. Members include employers, state licensing boards, educators, federal agencies, test 
developers, consumer groups, certifying agencies and professional associations. 



NCCA criteria for being eligible for review include: 

► The program shall be nongovernmental unless the certification is for government employees; 

► Must be national in scope; 

► Program must be operated by a not-for-profit agency; 

► The program shall have completed at least two national examination administrations prior to 
seeking NCCA approval; 

► Program must be administratively independent; and 

► There must be alternative pathways for individuals seeking certification. 
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EDUCATION-DRIVEN SKILL STANDARDS SYSTEMS 



The industrial changes of the past 20 years have pushed forward a process of change in the skills 
required of workers, which has brought changes to skill standards setting. Starting in the seventies and 
picking up speed in the eighties a new competency based system emerged. 

Occupational skill standards have been established primarily through the efforts of state level 
vocational-technical education. The 1990 reauthorization of the vocational education act, popularly known 
as Perkins II, provides for the first time fiscal support for national occupational standards as well as 
encouraging further state work through requirements for performance standards and Technical Preparation 
programs. 

Several consortia of states have been formed over the years to encourage collaboration on 
occupational skill standards. In this section, occupational skill standards will be used synonymously with 
competencies. Tasks will be referred to as components of skill standards. 

The educational development of occupational skill standards has been driven by the specific needs 
of industry. Vocational-technical education's primary approach has been to use occupational skill 
standards to develop task lists as the basic units of curriculum, instruction and evaluation criteria to ensure 
that students acquire the needed skills for jobs. The study found that approximately 700 committees using 
industry volunteers exist across the country. 

The study team surveyed state vocational-technical education agencies to learn about their 
involvement in skill standards. The following summarizes those findings. 

Purposes for development: 

► Curriculum development -- 48 states; 

► Guidelines for programs or courses -- 46 states; 

► Assessing student mastery -- 29 states. 

Initiators: 

In most states, education was the originator of the development of occupational skill standards. 
Only in seven states was business and industry cited as an originator. 

Influence on course of study: 

► Articulation between secondary and postsecondary programs - 47 states; 

► Development of course syllabi - 44 states; 

► Program certificates of mastery - 36 states; 

► Testing and assessing skills acquired - 42 states. 
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Partnerships: 

► Asking industry managers to help form occupational skill standards committees 41 states; 

► Using incumbent workers or supervisors in the standards committees — 43 states; 

► Using instructors from the occupational field in the study - 40 states; 

► Including academic instructors - 28 states; 

► Using curriculum developers to work with occupational skill standards ~ 35 states. 
Financing development: 

► Federal vocational education funds - 49 states; 

► State funds in addition to federal - 34 states; 

► Business and industry funding - 16 states. 

States have spent widely varying amounts on occupational skill standards development. The range 
is between $3,000 and $20,000 per occupational cluster. Approximately three quarters of the states are 
committed to funding ongoing maintenance and revision of occupational skill standards and related task 
lists. Most state development occurs as part of a consortium, such as the Vocational Technical Education 
Consortium of the States {V-TECSL with member states regularly adding to the pool of standards and task 
lists. Each year updating is required in some areas, particularly in occupations with changing technologies, 
such as printing, electronics, manufacturing, automotive and health specialties. Keeping those lists up to 
date is one of the greatest concerns of education and its partners. 

No one set of skill standards has been established for all states or is used by every state. 
Extensive lists of occupational skill standards are available across the nation. However, only 26 to 32 
states use a common set of standards for any one occupation. 

Starting point for standards setting and occupational analysis: 

► V-TECS catalogs - 38 states; 

► Consortium lists and materials - 33 states; 

► Business and industry lists - 37 states; 

► Previous state and local standards - 30 states; 

► Apprenticeship lists — 21 states. 

Process for development of standards: 

Most states use a process involving input from workers in jobs for which the standards are being 
developed. The most common is DACUM or modified DACUM - 34 states; other industry methods - 29 
states; and V-TECS or modified V-TECS - 26 states. 
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Methods of assessment used: 

► Written cognitive tests ~ 41 states (only half using computerized versions); 

► Simulation -- 26 states (with 23 using situational tests); 

► Performance tests - 38 states; 

► Combination of above - nearly half. 

Most states also include in their assessments of vocational-technical students mastery of academic 
competencies in math, communication, science and reading as well as employment related skills such as 
problem solving, resume and application, and human relations on the job. 

Use of the standards: 

The greatest usage of the occupational skill standards is by local districts. Partnerships with 
business, industry, apprenticeship and Job Training Partnership Act (JTPA) programs provide an 
opportunity to share occupational skill standards as a common base for skill development and employment 
practices. 

Strong links are emerging between occupational skill requirements for individuals and performance 
standards for vocational-technical education programs established under Perkins II. Thus far, 33 states 
require local programs through performance standards to use occupational skills and competencies. 

Barriers to be addressed: 

A number of obstacles must be overcome to take full advantage of the work done thus far within 
vocational-technical education to assist development of a national voluntary system of skill standards. 
Major ones include: 

► Assuring that the instructional staff (both academic and vocational) is sufficiently knowledgeable 
of the needs of industry; 

► Development of a more sufficient and affordable assessment system that carries with it the 
portability deemed of value by industry; 

► Encouraging expanding usage by more local school districts and other education and training 
providers. 

Volume II of the report contains nine case studies of state activities regarding occupational skill 
standards as well as the complete survey results from all 50 states. 



In addition to their work with vocational-technical educators, industry is involved through other 
venues in the skill standards issue. The focus of this section is on the role of national industry and 
occupation focused organizations. It is important to note at the outset that "skill standard" has 
traditionally been a technical term, not a policy term. Policy discussions tend to use the term 
"credentialing", and what follows discusses the issue using that term. 



INDUSTRY DRIVEN SKILL STANDARDS SYSTEMS 
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Credentialing activities can include: 

1 . Prescribing education and experience qualification for certification candidates; 

2. Establishing for potential accredited institutions qualifications for curriculum, faculty, and 
facilities; 

3. Administering competitive exams; and 

4. Conducting assessment visits. 

Industry groups have taken on these actions in many sectors as a self-regulating, privately-driven 
action. The essence of the arguments for such actions is that self-regulation prevents costly and 
cumbersome government regulation and that the privately-driven approach requires the providers of the 
service to stay close to changes in the marketplace. 

Self-regulation of credentialing is very much an American strategy. By far the most typical pattern 
in other industrialized countries is for government to perform the credentialing functions. 

This study focuses on occupations not normally requiring a four-year college degree or more as 
minimum entrance requirements and on national trade associations that directly provide or indirectly 
sponsor a certification service. The journeyman card represents a form of certification that does not fit into 
the trade/professional association category. However, the apprenticeship and upgrading programs jointly 
sponsored by labor/management committees were considered part of the industry driven skill standards 
development "system" for purposes of this study. The research does not include firm-specific skill 
standards or apprenticeships. 

Volume III of the report contains the detailed descriptions of each industry-based certification 
system our research uncovered. The following highlights some of the key findings. 

Gaps and Coverage 

There are major industry sectors that do not have a tradition of promoting industry-wide skill 
standards such as agriculture, mining, retailing and large portions of manufacturing, in manufacturing, 
firms are often involved in firm-specific apprenticeship training programs for which the course of study and 
certification may or may not reflect cross enterprise standards. See Table 2. 

By far the largest number of certification programs are directly related to occupations and 
industries where there has been (a) intervention by government to regulate the industry or (b) the threat of 
government regulation. Certification of individuals has been chosen as the route for self-regulation by the 
affected industries. Health care and real estate are examples of the first type and direct selling is an 
example of the latter type. 

Most certification programs reflect specialties or market niches. 

The Role of Industry Associations 

In very few instances was there found a single industry association that represented the total 
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industry, offering the only certification program used by all of that industry. This isn't surprising, given 
that mcs: industry associations are themselves specialty organizations. 

There is a strong pattern of influence that emanates from a primary profession such as 
engineering, accountancy, or medical over the certification programs that recognize the technical workers 
within the same field. For example, many of the allied health fields are clearly modeled after the medical or 
dental preparation programs for doctors. 
Characteristics of the Certification Programs 

In many ways the current state of affairs means that the old adage "let the buyer beware" holds 
sway when discussing certification and credentialing programs. This observation is made not to pass 
judgement on any organization or institution, but simply to note that one cannot assume that a person 
claiming to be certified or credentialed has completed an assessment of skills and knowledge in a program 
that has validated the standards or the assessment instruments. 

The Patterns of Practice 

There are some common patterns in the certification systems: 

1. Most programs offer only one or two recognitions; 

2. Most are linked to time in the workplace; 

3. Most have alternate paths that allow candidates to credit schooling with less time spent in the 
workplace to qualify for assessment; 

4. The overwhelming model is to assess knowledge through passing a paper and pencil test; 

5. Most have some form of required recertification based on continuing education and professional 
development activities; 

6. Most have permitted a time limited period for "grandfathering" members of the profession 
without requiring the individual to sit for exams; 

7. Most have developed a core body of knowledge that is required in order to minimally qualify for 
being considered a candidate; 

8. Some have established linkages with recognized organizations such as the American Council on 
Education in order to have the passage of an exam count for college credit. 

Management and Financial Issues 

It is often the case that non-profit organizations are established by a single trade association or, in 
some cases, by several trade associations to support research, development and certification program cost. 
In some instances, these non-profit organizations have the responsibility to set product standards as well 
as establish skill standards for individual certifications. 

Recertification is becoming a larger issue for many of the programs as a result of technology 
changes, yet many have found members resistent to the requirement that some form of testing be part of 
the continued recognition. Without the passage of some test and othet assessment of skills it is necessary 
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for the sponsoring certifying agency to keep records of proof that the individual has complied with 
recertification requirements. 

In any voluntary certification program, promotion is a management task. Individuals must be 
encouraged to become participants and employers must be encouraged to utilize the certification in their 
hiring and promotion practices and policies. 

One of the most impressive features of certification programs is the willingness of volunteers to 
contribute substantial time and energy to support the effort. However, volunteers cannot do it all. Most 
programs are able to eventually become self-sustaining by charging a variety of fees for different parts of 
the certification program. 

Speed of development is directly related to whether up-front seed money is provided by the parent 
trade association or professional society. 

The total cost varies, but several hundred thousands of dollars are required (including voluntary 
time of experts) to launch a substantial and credible program. 



Program Recognition or Accreditation 

One of the stated goals of the federal government's initiative is to promote a competency-based 
system that builds into the education and training curricula the workforce skill and knowledge requirements 
identified by industry. This goal, if realized, will represent a sharp break with current practices, which 
focus largely on input measures in combination with time-of-study measures, not competency attainment. 

The typical pattern promoted by industry when supporting accreditation programs has been to 
develop agreement among educators and themselves regarding what is often called the common body of 
knowledge required to practice in a profession, and to then ensure both that courses contain that 
knowledge and that the educational institution provides qualified faculty, adequate equipment and other 
key inputs. Accreditation usually begins with a self-assessment process followed by a peer review 
organized by the sponsoring body. 

Apprenticeship programs are similarly focused around a common body of knowledge, type of 
equipment and qualifications of instructors — all input measures. 

Both of these traditional quality assurance forms also have in common a program that is essentially 
time-based. 

There are only limited examples of national organizations that are attempting competency-based 
approaches now. However, it is instructive to note that a substantial number of industry trade associations 
that are deeply involved in sponsorship of accreditation programs are also involved in a certification of an 
individual's competencies in a separate program effort. As one individual interviewed stated succinctly, an 
accredited program only allows an individual to learn, while the certification testing process finds out what 
they learned. 
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Apprenticeships 

The costs for the apprenticeship model of credentialing is often supported by a self-imposed tax 
that employers pay as a result of collective bargaining agreements or by the choice to sponsor such a 
program. In most joint union/employer sponsored programs, apprentices do not bear any of the related 
training costs nor are testing fees required. 

There are no requirements that an approved program include such quality assurance mechanisms 
as testing or performance measurement and that has, in some instances, caused problems in the 
"portability of the credential" from one community or state to another. 

Concern about Instructors 

The qualifications of instructors is a theme of concern that permeates almost all of the industry 
associations that are involved with any form of credentialing. 

Several of the national apprenticeship programs invest considerable time and energy in preparing 
experienced industry workers to perform the all-important task of providing the related instruction. Many 
of the industry certification programs have developed a screening mechanism to vouch for the quality of 
instructors that can be used in their own education programs. 

In some cases, educational tenure policies are sharply criticized by members of industry who are 
not convinced that ail of the instructors have "kept up" with the changes in the occupation or industry. 

The quality of work site instruction is a related and important issue. Often, the individual who is 
the work site supervisor must be "certified" themselves in order for the student to gain the necessary 
credits for graduation or progression. 

Relationship to Licensing 

Complex is the operative word regarding the relationship of licensing and voluntary credentialing 
systems. It is often the case that associations representing a particular occupation aggressively seek the 
governmental blessing of requiring licensure as a way of controlling entry into the practice of the 
occupation or to help members address potential health and safety liability legal claims. 

At the same time, there are obviously mixed opinions about the role of government, often resulting 
in contentious debates. 

It is worth noting that many voluntary certification and accreditation organizations have close 
working relationships with state licensure boards. 

Unsettled and Continuing Issues 



Important issues emanating from examining the current industry credentialing practices include: 

► There is no common framework or language between and among the programs and the education 
community at large, let alone among the general public. 

► Few credentialing programs are targeted at the entry-level workforce. 



ERIC 




19 



► Competition among professional associations, certifying/accrediting groups and even licensure 
bodies is a concern. An example: 21 organizations offer nursing certification, a proliferation that 
has led to some duplication. 

► Unwillingness of specialized accreditation bodies to recognize either course work in a related field 
or experience gained on the job. 



The study team reviewed the skill standards systems in six countries representing a range of 
approaches: Australia, Canada, Denmark, Germany, Japan and the United Kingdom. These tend to break 
into three categories: 

1 . The "initial preparation" model represented by Germany and Denmark focuses on the school- 
to-work transition for young people. 

2. The "craft certification" approach represented by Japan and Canada meets the needs of more 
mobile workers. 

3. The "comprehensive" model found in the United Kingdom and Australia is the youngest and still 
emerging category. 

In looking at these six countries, three important differences between the United States and all of 
them should be noted: 

Gap 1 Most other countries are more advanced than the U.S. in terms of supporting education 
and/or work-based skill standards systems. 

Gap 2 In each of the other countries studied an exit from compulsory school examination system 
is in operation, often used to sort people, a strategy the U.S. has eschewed. 

Gap 3 - In the other nations, there has been a long history of the central government supporting 
and promoting a third party certification of skills and knowledge gained through the vocational preparation 
process, including formal mechanisms to involve industry and employee representatives. 

Volume IV of this study describes all six nations' strategies in some detail. Listed below are some 
key findings. 

Initial Preparation Models - Denmark and Germany 

Both of these nations' frameworks are school-to-work transition models, and both have long 
histories of involving industry in the education and training of the workforce. Both have been firm 
regarding the critical role of industry in establishing standards and in designing and implementing 
certification of skills. 

The role and structure of national standards in the basic vocational education and training system 
in Denmark offers an interesting model for U.S. consideration. The German Dual System is a leading 
international example of a school-to-work transition system based on national skill standards. 



OVERVIEW OF SKILL STANDARDS SYSTEMS IN SELECTED COUNTRIES 
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Craft Models - Canada and Japan 

The critical feature that ties Canada and Japan together within the skill standards systems is the 
focus each has placed on the development of skill certification systems for the occupations that have a 
substantial amount of mobility across jurisdictions — the crafts and trades. 

Both countries have "wrap-around" support systems for the entities that have the lead 
responsibilities for occupational preparation of the workforce. However, due to fundamental differences in 
the placement of the center of responsibility for training the workforce -- in Japan it is the employers and in 
Canada the individual provinces -- the central government support systems have different design 
characteristics. 

National skill standards in Japan are driven largely by government and employer organizations, and 
focus heavily on incumbent workers Canada excels at establishing both a common language and portable 
credentials. 

Comprehensive Models - United Kingdom and Australia 

Clearly the most ambitious and, perhaps radical, of the skill standards systems can be found in 
English speaking countries within the United Kingdom and the Commonwealth member state, Australia. 

In both instances, prior to the 1980's the traditional role of the central government could be 
characterized as laissez-faire in the arena of promotion of skill standards. During the past decade, 
however, there has been a dramatic shift. In these countries, the promulgation of skill standards and 
certification systems are viewed as central to the competitiveness strategy of the overall economies. 

The development of industry driven skill standard systems are having direct impact on the 
organization and structure of vocational preparation institutions. Also, the traditional academic education 
systems are experiencing changes due to the establishment of the skill standards systems. 

Australia and the United Kingdom offer comprehensive models that have many parallels to the 
process emerging in the United States and the *ore contain relevant framework and technical lessons to 
be considered. 

Lessons for Developing Skill Standards Systems 

There are a series of lessons that one can discern from investigating the standard setting process 
in other countries: 

► An open and clearly understood process is needed to conduct consultations among the 
stakeholders; 

► The government has a powerful role to play, particularly as an advocate and promoter of the 
process; 

► Industries must be responsible for developing the standards; 

► Fiscal support from the government is required to assist the industries in the development of 
standards that are to have nationwide applicability; 
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► The standard setting process is facilitated by creating a standardized language and format that 
must be followed in creating national standards; 

► The standard setting process is, in fact, a political and negotiated process; 

► There must be education led processes established to facilitate the transformation of standards 
into curriculum; and 

► The standards generate the need to realign intergovernmental systems. 

RECOMMENDATIONS AND CONCLUSIONS 

All of the research leads to a question: What does this imply for the further development of a 
voluntary skill standards system in the United States. The U.S. Department of Education asked the 
research group to explicitly answer six questions to help understand the scope of the roles for the federal 
government and the private sector in the development and implementation of skill standards to improve 
U.S. competitiveness in the global market. The six questions are: 

1 . What roles should industry and the federal government play in developing a nationwide system 
of skill standards? 

2. What arrangements must the federal government adopt to encourage industry to become 
involved in the development and use of skill standards? 

3. What are the best forms of assessment/testing to use in the skill standards systems? 

4. How could a nationwide system of skill standards affect education programs at state and local 
levels concerning educational program improvement, curriculum modernization, and educational 
choice? 

5. What testing guidelines should be recognized to insure against discrimination based on gender, 
ethnicity or disability status? 

6. To what extent do the current apprenticeship and cooperative education systems in the U.S. 
provide a framework for setting skill standards? 

RECOMMENDATIONS 
The Ideal Approach 

The suggestions that follow are rooted in assumptions about an "ideal" client-centered skill 
standards system where individuals and employers are considered the client. The specific assumptions are 
the system would: 

1 . Not be bound by individual student/workers age and would be widely accessible to young 
people and adults; 

2. Not be bound by type of institutions providing education or training and flexible in response to 
changing, and sometimes different, needs of individuals and localities through a variety of forms 
(e.g., education, full-time and part-time training); 
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3. Would be able to meet the needs of individuals regardless of type of education and training they 
are pursuing (e.g. initial preparation, continual, upgrading, or remedial); 

4. Would allow career paths within and between industries; 

5. Be explicit so that firms, education, training providers, and individuals know what the standards 
are and where information about them can be obtained; 

6. Be a competency-based system; 

7. Have a formal assessment and certification of an individual's skills documented by a third party; 

8. Be a progressive system so that people can build upon blocks of competencies and adapt to 
technological and market changes, to improve their prospects or to explore their potential; and, 

9. Have a common framework and use common language when describing skill levels across 
industries and occupations in order for both individuals and employers to easily understand the 
expectations and value of the use of the standards. The framework should progress from initial 
(entry) qualifications through several levels to mastery and/or specialization recognition. 

These criteria provide the starting point for our responses to the Department of Education's 
specific questions. 

QUESTION 1: WHAT ROLES SHOULD INDUSTRY AND THE FEDERAL GOVERNMENT PLAY IN 
DEVELOPING A NATIONWIDE SYSTEM OF SKILL STANDARDS? 

Industry must be in the recognized lead role for the development of the standards and the federal 
government must help facilitate such development. It is essential to the success of any skill standards 
system that the employer community recognizes the value of these measures to their economic success 
and that they accept them as theirs. 

The federal government has the critical function of establishing a consensus building organizational 
structure. This function cannot be overemphasized. Each of the other countries learned how essential it is 
to have quasi-independent organization(s) that are charged with the responsibility to bring together the 
various stakeholders. 

The next steps that need to occur to move the development of a voluntary skill standards system 
forward includes: 

► Establish a framework for generating valid and reliable skill standards, assessment and 
certification systems. 

Common nomenclature and definitions are essential in order to facilitate the expansion and use of 
voluntary skill standards. There must also be agreement on the best ways to describe and assess 
performance levels for the complex skills required in the workplace. 

» Establish common levels of qualifications. 

The U.S. will need a progressively complex set of levels of knowledge and skills mastery required 
for individual career entry and progression. 
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► Establish procedures for benchmarking to international standards and for upgrading and 
maintaining the skill standards continuously. 

Skill standards will only lead to higher skills and assist in the improvement of the economic position of 
firms and individuals if they are benchmarked against the highest international standards. 

► Establish criteria and recognition procedures for organizations that develop standards. 

The criteria should include assurances that any such organization recognized represents all aspects 
of the industry, that geographic representation exists and membership includes incumbent workers. 

► Establish criteria and recognition procedures for organizations which design and award the 
certificates and insure the quality control over the assessment systems. 

This recommendation is predicated on two assumptions: (a) a third party assessment process is a 
component of the skill standards effort, (b) several different organizations can and will be involved in such 
assessments. 

► Establish procedures that ensure the security of the individual records of persons that have been 
awarded recognition of having achieved the qualiflcation(s). Also develop a system that will ensure 
easy access to the standards themselves. 

The report of findings from assessments needs to be generated in a form that is easily recognized 
and is portable from one place to another across the country. There is also a need to develop and maintain 
a common locator system for all of the standards. 

► Establish procedures for disseminating skill standards to employers, education and training 
providers, assessment systems, and individuals. 

Voluntary skill standards have meaning when they are widely used. Considerable attention must be 
given to promoting use of the standards. Such promotion should be rooted in fact and detail regarding the 
benefits to individuals, employers and education and training providers. 

► Develop a shared development and maintenance funding strategy. 

The development of comprehensive and nationally applicable skill standards will require support 
from both the pubic and private sectors. This support can take many forms but without it nationwide skill 
standards will not exist. 

QUESTION 2: WHAT ARRANGEMENTS MUST THE FEDERAL GOVERNMENT ADOPT TO ENCOURAGE 
INDUSTRY TO BECOME INVOLVED IN THE DEVELOPMENT AND USE OF SKILL STANDARDS? 

Note: For purposes of this discussion the term industry is meant to include employee organizations 
as well as employers. 

Millions of volunteer hours are donated each year in the development of validated skill standards. 
We see that from both the education and industry driven skill standards programs across the country. It is 
hard to imagine that it will be difficult to garner industry involvement in the development of the standards. 
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Financial support will be needed to encourage the development of comprehensive skill standards that will 
assist the novice as well as the specialist/master. Ensuring a comprehensive system, is we believe, one of 
the key reasons for government action. 

► Demonstrate the value of skill standards to encourage use by employers. 

It is to the second part of the question that our study suggests substantial attention is required. 
Use within industry may prove to be problematic. There needs to be an aggressive "show them and help 
them" strategy to encourage industry to use the skill standards systems. 

► Develop support materials to focus use of skill standards in the workplace. 

In addition to "show them" strategies a great deal of "help them" work has to be considered with 
a strong focus on the front line supervisors and managers. 

► Provide leadership and support to help employers comply with civil rights laws. 

A major barrier to implementing standards by employers is the fear of civil rights violations. 
Substantia! support and development of technical and training materials must be provided that will advance 
the cause of equity, access and quality standards simultaneously. 

► Promote the use of skill standards through incentives. 

Additional incentives for future consideration include the shifting of the treatment of training in the 
tax code from one of a current expense to an investment or promoting the creation of grant/levy incentive 
plan based on a portion of payroll expenses. 

QUESTION 3: WHAT ARE THE BEST FORMS OF ASSESSMENT/TESTING TO USE IN SKILL STANDARDS 
SYSTEMS? 

There is no silver-bullet answer to this question. No one form of assessment or testing was 
identified as being superior either in the literature search or in the in-depth review of selected industry- 
based skill driven certification programs. 

It is important to note the criticality of first performing a valid and extensive job analysis for the 
creation of standards. Only through a systematically performed job analysis can one insure that the 
standard^ and aspects of the certification program accurately represent the needs of the industry and the 
work of the job incumbent. Secondly, provisions of the Civil Rights Act, the National Labor Relations Act, 
and other appropriate law associated with assessment and job analysis must be followed. 

► Establish consultation with experts to study and improve assessments processes. 

Materials collected from trade and professional associations for this study could be used to identify 
organizations using a range of assessment instruments and criteria to award recognition. Based upon such 
a review, a series of consultations should occur with experienced individuals to help inform decisions about 
what are the most feasible and appropriate next steps that would help to establish more valid assessment 
techniques. 
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► Federal agencies should develop a assessment research strategy that links several efforts 
together. 

Promotion of a voluntary skill standards systems is a part of the response to the National Education 
Goals, specifically, Goal 5. The offices and agencies within the federal government that have responsibility 
for the development and implementation of the various assessment efforts embedded in that goal should 
coordinate their research and development activities. This must include assessments efforts tied to school- 
to-work initiatives. 

QUESTION 4: HOW COULD A NATIONWIDE SYSTEM OF SKILL STANDARDS AFFECT EDUCATION 
PROGRAMS AT STATE AND LOCAL LEVELS CONCERNING EDUCATIONAL PROGRAM IMPROVEMENT, 
CURRICULUM MODERNIZATION, AND EDUCATIONAL CHOICE? 

If care is taken to involve all of the stakeholders and if the necessary support systems are 
developed, the impact on the nationwide system will be profound. 

There will need to be a clearly developed consensus regarding the appropriate roles and 
responsibilities of education and training providers in the development and implementation of a skill 
standards system. 

This study, we believe, highlights the fact that the most important functions to be completed by 
the education and training institutions are the delivery of education and training services, not the 
development of the standards themselves. Yet, without strengthening the capacity of the education and 
training system the skill standards "system" will flounder. There will be a need to develop an array of 
capacity building networks to assist education and training providers. 

► Develop an information dissemination strategy to explain the intent and potential of skill 
standards. 

At this point in the history of skill standards, one of the tasks is to help people understand what is 
meant by skill standards and how they can help improve the education and skills preparation of individuals. 
Several parts of the U.S. Departments of Education and Labor need to be involved and one of the most 
efficient strategies is to use the network of national associations meetings and publications. 

► Develop and assist the implementation of technical assistance to help promulgate the utilization 
of the standards. 

Consortium-type efforts involving one or more national networks may be a cost-effective approach 
to develop materials related to the use of the standards. For example, it may be desirable to develop a 
project that would include representatives of state departments of education, community colleges and 
vocational technical education to provide technical assistance to their member institutions regarding how to 
develop articulation agreements between secondary and postsecondary institutions that would incorporate 
competency based instruction, recognition of work-site learning and independent assessments of 
student/worker skills and knowledge. 




► Develop and assist organizations that would have responsibility for being the "linking 
institutions" between the standard setting and assessment bodies and the education and training 
provider community. 

Focused, sustainable, and jointly "owned" institutions we believe will be necessary. For lack of a 
better descriptor, we have called these, linking institutions. There are several approaches that could be 
considered for the development of such institutions. For example, organizations could be established 
around major occupational clusters or geographic regions. The essential point is that industry 
representatives, state governments, and most particularly the representatives of secondary and 
postsecondary institutions must come together and agree upon a common agenda that will continuously 
translate skill standards into curriculum, update curriculum, instructional materials, and make it widely 
available to all types of education and training institutions. 

► Address the issue of staff development within the education and training system. 

This issue was consistently identified as a concern. Instructors from all types of institutions need 
to have a deep understanding of what is happening inside workplaces. Finding qualified instructors will 
continue to be one of the most important barriers to ensuring quality education and training. 

► Develop improved program quality assurance mechanisms. 

The study highlights how essential it will be to link skill standards effort with any effort to accredit 
or officially recognize a program of study within any education or training institution. 
The federal Department of Education along with the accreditation community and representatives of state 
government need to be substantially engaged in consultations that focus on how to develop improved 
quality assurance processes for recognizing programs of quality that will meet the needs of employing 
community. Additionally, federal and state governments should commit to incorporating skill standards 
their workforce development programs performance management systems. 

QUESTION 5: WHAT TESTING GUIDELINES SHOULD BE RECOGNIZED TO INSURE AGAINST 
DISCRIMINATION BASED ON GENDER, ETHNICITY OR DISABILITY STATUS? 

► Build upon the guidelines that have already been established by private, voluntary national 
standard setting quality assurance organizations. 

There are two organizations that have developed strong and well researched testing guidelines for 
use in standard setting and assessment programs; the American National Standards Institute (ANSI) is the 
U.S. member body of the International Organization for Standards (ISO) and the National Commission for 
Certifying Agencies (NCCA). The guidelines vary slightly, probably reflecting the history and traditions of 
the organizations themselves. Both organizations' guidelines, could be built upon, to assist any testing 
efforts to ensure against discrimination based on gender, ethnicity or disability status. 
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QUESTION 6: TO WHAT EXTENT DO THE CURRENT APPRENTICESHIP AND COOPERATIVE EDUCATION 
SYSTEMS IN THE U.S. PROVIDE A FRAMEWORK FOR SETTING SKILL STANDARDS? 

Neither the registered apprenticeship system nor the cooperative education system, in and of 
themselves represent a framework for setting and implementing skill standards. However, both systems 
provide lessons for the new system from their implementation history, and both may be regarded as 
potential points of articulation between systems currently administered by the Departments of Labor and 
Education and the new skill standards system. 

The vocational-technical education system, within which many of the co-op programs are 
embedded, provides a larger context within which to discuss the development of the framework for setting 
skill standards. 

The Registered Apprenticeship Programs 

The federal Bureau of Apprenticeship Training's (BAT) regulations predominately focus on process 
and program structure issues and the only major "quality assurance" standard contained in those 
regulations relates to wage rates. The standards expressed in the registration document and included in 
the individual apprenticeship agreements that serve as contracts for employment are most often time 
based, not competency based. Attention is focused almost exclusively on the initial preparation of workers, 
and little attention is given to the continual upgrading of the workforce and very few stress the entire 
career path. Furthermore, the process standards do not require continual updating and validation of the skill 
standards, nor do they promote the use of the best practices in assessment. The emphasis placed on 
assuring quality instruction by many of the national BAT programs appears to be of vital importance in the 
conduct of a quality apprenticeship program, however, the BAT regulations are silent on the issue of 
instructor qualifications. 

However, individual BAT-recognized apprenticeship programs do provide excellent examples of 
how to link industry skill requirements to related instruction and then certify individuals based on nationally 
recognized third party assessments. 

Cooperative Education Programs (Co-Op)/ Vocational-Technical Education System (Voc-Tech) 

Some cooperative education programs utilize skill standards in their agreements between business, 
education and student, however, it is not one of the drivers in the standards development process. 

The "parent" authorizing legislation for the co-op education programs, particularly for those at the 
secondary level, has traditionally been through the vocational-technical system. It is important to discuss 
how this system can support the development of a national skill standards system simply because it will 
have a broader impact than that of the co-op education program. 

It is probably a safe assumption that a substantial burden of implementing the educational 
components of the occupational preparation for the workforce will fall upon the secondary and post- 
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secondary institutions that is commonly called the voc-ed system. Clearly, the work within the states, to 
date, to establish skill standards and related competency based curriculum for occupational related 
education has been accomplished primarily through the voc-ed structure. Als? the challenge to improve the 
articulation between secondary and post-secondary education institutions has been spurred on by the 
establishment of the Technical-Preparation program within the Perkins II legislation. The effort to increase 
the integration of academic and vocation education related curriculum is also being stimulated by the 
Perkins li legislation. 

The study documented that the primary use of occupational skill standards has been to develop 
task lists as the basic units of curriculum, instruction and evaluation criteria for entry to sometimes 
"journeyman" levels. The study indicates that employers have been willing to work with the state officials 
on all three of these tasks. However, it also reveals the additional opportunities need to be pursued within 
the vocational and technical education system: 

► Have states collectively examine the current base of occupational skill standards. 

Collective work among all of the states could be undertaken to look across all major occupational 
areas for which standards have already been developed to ascertain where it would be possible to (a) 
identify common standards that could be used within a national framework using common language and 
competencies, and (b) assess where it would be possible to expand the areas of identified occupational 
competencies {e.g. there were 120 identified for agriculture and 62 for health). While numbers and list do 
not tell the whole story they are an indicator of level of effort that may not mesh with the highest needs of 
the labor market. 

► Support an R&D effort to identify and measure the competencies that can best be learned within 
the workplace versus the classroom. 

Attention is needed to sort out which types of competencies are best learned at the work-site, 
under what conditions, versus what competencies are best learned in a classroom or a laboratory. This 
research should be linked to testing new forms of evaluations and assessments. A part of the focus should 
be to determine what will generate a tighter link between the employers and classroom instruction. It 
should also be linked to testing and developing "portable credentials". 

► Improve the pipeline for assuring high quality professional capacity. 

Work among the institutions of higher learning, the vocational education research community, and 
state policy makers is needed to help the vocational education community to continue to improve and 
expand competency based curriculum and instructional techniques. Work should also include developing 
improved student assessment methods that could be incorporated into a commonly recognized portable 
certificate effort. 

► Mesh the use of Skill Standards with the Development of Performance Standards for Vocational 
Education. 

The Perkins II legislation requirement that states develop performance standards for secondary and 
postsecondary programs should, overtime, include the use of skill standards. 
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ADDITIONAL OBSERVATIONS 

There are additional observations that were not explicitly required to be addressed in the contract, 
but emerged in the study that merit consideration by the federal government. 

► Use the common language of standards in classification systems. 

It will be important to pay attention to the issue of using common language across several 
standard information systems such as the classification of occupations and industries, (i.e. the Dictionary of 
Occupational Titles, the Standard Occupational Classifications and the Standard Industrial Classification, 
etc.) 

► Inform the general public and support enhanced counseling services. 

A substantial effort must be made to help individual students and their parents learn about the skill 
standards and their implications. Individual counselors responsible for helping students in a wide variety of 
institutions should have a deep understanding of the career and educational options that will be derived 
from meeting the standards. All career information systems should incorporate information on skill 
standards and credentials. 

CONCLUSIONS 

The largest challenge, the team believes, will be the development of the capacity to exploit the 
standards to their fullest potential within organizations and governments. This will require developing 
consensus and common action plans among various parts of the federal government as well as within 
industry networks and sub-national units of governments. This will require concerted and coordinated 
action on the part of several federal Departments. 

The reality is that all of the processes, including the development of a common framework within 
which the standards are established, within and across industries, will be a politically negotiated process 
among the various stakeholders and will take several years to develop. This means that the structures and 
systems that will be established need to be sufficiently flexible so that continuous refinements and 
improvements will be encouraged. 

It is essential that a wide array of capacity building efforts be established that focus on the needs 
of education and training organizations if the skill standards initiative is to become a part of the human 
resource development system in the United States. Other countries, including those that have had 
traditionally strong federal government presence in setting education policy, have found it necessary to 
develop special support systems for the education and training providers to establish the appropriate 
linkages with the industries within this arena. The non-directive role that the federal (and arguably the 
state) government has over education and training institutions in this country makes it even more essential 
that substantial attention and careful thought be given to developing sustainable institutional arrangements. 
Agreement will need to be reached among the levels of governments responsible for supporting education 
and training systems regarding the best approaches to be pursued. 



A quiet and deliberate approach that is also aggressive will be required to generate the mutual trust 
between the private and public sectors, between the potential private sector partners, between the 
different levels of government, between different types of education and training institutions, between 
different agencies within the same federal departments and across federal departments. The IEL team 
came to believe the value of establishing skill standards was worth the challenge. 
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CHAPTER II 



INTRODUCTION AND THE CONTEXT 



FOR ESTABLISHING SKILL STANDARDS IN THE UNITED STATES 



A. BACKGROUND 

There is a wide ranging set of historical conditions, some deeply rooted in the American culture, 
that have influenced the relationship between business and education institutions in setting skill standards. 

The following represents a cursory overview of both historical and recent developments influencing 
the evolution of more formal systems between national organizations representing employers and the 
various parts of our nation's education system. These factors may appear to be unrelated to one another 
and indeed we have often treated them as though they are. In combination, however, they represent some 
of the confluence of forces and factors that must be taken into account when reviewing the landscape in 
any study of the linkages between business and education developing and using common skill standards. 



► Our country's early support for universal compulsory education, including vocational preparation 
in the early part of this century, reflected the preferences of the society to at least provide a free 
basic education for our citizens to making their way into the world of work. The high school 
diploma became the basic passport for continued education and the means of access to the more 
"attractive" jobs. Unlike other industrialized countries, the U.S. never developed a national system 
of formal linkages between employers and schools to facilitate the transition of young people from 
school to work. 

► The goal of obtaining at least a high school diploma has long been thought by many to be the 
most important passport to doing well in the work world and by most measures this has proven to 
be correct. But, as an ever increasing number of individuals obtain diplomas, the value of the 
diploma in fact diminishes. Additionally, it has been found to be an inadequate "signaling device" 
because it has proven to be unreliable in indicating to the employer community whether the holder 
of the diploma has achieved necessary skills. Also, due to changing workplace requirements, calls 
are coming from many quarters for different forms of certification of knowledge gained and skills 
acquired. 

► One of the more important developments over the past three decades is the growth of public and 
private postsecondary institutions with explicit missions to provide specific occupational skill 
training. Most of these community colleges and technical institutions have developed strong 
networks with local employers in order to provide initial preparation, upgrade training and, when 
necessary, provide remedial education services. The Technical Preparation portion of the Perkins II 
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legislation has had a significant impact on the relationship between these postsecondary 
institutions and the employers with whom they work. Commoi.iy called Tech Prep, key features of 
this program include the requirement that planning occurs among secondary and postsecondary 
institutions and business and industry. The process is focused upon the needs of local areas and 
the goal is to establish articulation agreements among the "partners." An example of some of the 
questions that this new Tech Prep approach requires the partners to answer include the following: 
Are skill and quality standards unique to the local area? Will there be formal agreements forged 
between the education partners and individual businesses and industries on requirements for entry 
and advanced levels of work? How will skill standards be measured locally? How will 
transportability of acquired skills be accomplished? 

► Defining the content of education credentials reflects a unique American approach. Unlike other 
countries where the central government has a strong role in defining education standards for the 
compulsory school years, in most parts of this country until rather recently, defining the content of 
the general education credential was left largely in the hands of local boards of education. 
Currently, however, states are using their authority to establish more common "standards" before 
the most basic credential (high school diploma) can be issued. 



At the postsecondary level this country has turned to the private sector to regulate itself. The 
quality assurance standards have been developed through a complex network of accrediting organizations 
that have, in turn, used a voluntary commission to establish the criteria for determining how to award 
credit and credentials. 



► With the acceleration of technological change and international competition, initial preparation, no 
matter how good, is not enough. Continual preparation is increasingly important. There is no 
consistent way of documenting or certifying the skills acquired by adults on the job or through 
education and training programs offered by a variety of public and private providers. Without 
being able to document knowledge and skills acquired in a common language acceptable to 
employers and educators, adults have trouble building a portable credential that acts as currency in 
both the employment and education sectors. 

► Because education has been viewed as the great "equalizer," the importance of access to 
education has shaped much of the federal rcle in education over the past several decades. While 
no one would assert that all of the issues regarding equity and access to education are totally 
resolved in this nation, there has been an increasing recognition that just equal access, alone, is 
not sufficient for the labor market of tomorrow. There is a growing consensus that legitimate 
national concerns about the quality of education and training programs supported by the federal 
government must be addressed. Any discussion of quality leads to a discussion of definitions. 
Discussing definitions leads to discussing the providers of definitions. These discussions have, in 
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part, led us to the current endeavors to establish more efficient, effective, and sustainable linkages 
between industry and a wide array of educational institutions at the secondary and postsecondary 
levels. 

► More than most other countries the equity concerns have also influenced other federal actions 
that must be considered in any certification of knowledge gained and skills acquired. The Supreme 
Court has on several occasions called into question the testing and assessment instruments used 
by businesses to recruit, hire, and promote employees. Any assessment and testing system used 
cannot penalize an individual's right to the job based on race, sex or some handicapping 
conditions. 



The settlement patterns of our country, the rich natural resource base that undergirded the building 
of the economic wealth of the nation, the forms of production processes used to build the industrial base, 
and the free market economic system have all influenced the linkages between education and the 
workplace. 

► The traditions of guilds and societies were brought over from the Old World but they have taken 
on distinctly American characteristics. This has been due, in part, to the continual expansion of 
the westward movement of the polyglot mixture of immigrants making up the population. The 
dominant role that guilds and societies played in the Old World's politically powerful city/states 
never took root here, but an historical analysis will show thsy have had an influence. 

► During this century the need for occupationally specialized workers was minimized due to the 
strong influence of the Taylorist model of manufacturing. Following this model work was 
organized in such a way that the worker could learn the specific tasks within a few days or even 
hours. Job rotations were not the norm nor was there a concentrated push from unions during 
collective bargaining negotiations in the manufacturing arena for multi-skilled jobs and the 
attendant training required for such jobs. There are some obvious exceptions to this generalization 
such as the machine maintenance and precision tool and die making fields. Skill standards were 
normally set in these cases through the establishment of apprenticeship programs, workers often 
being selected for such training based on longevity on the job. These manufacturing practices 
helped develop a tenet that it was the job of schools to provide the basic education for workers 
and it was the job of employers to train the workers for the specific job. The effect of this has 
been that for entry level and line worker positions in most manufacturing and some of the service 
industries there has not been a strong history of skill standard and competency setting systems. 

► The military, one of the largest employers in the United States, has long been involved in 
establishing the skills required for each position in their service. Since training must be geared to 
ensuring adequate attainment of skills, efforts to classify tasks and competencies has been of 
critical importance. For many years the nature of the Gaining has been very specific, as have been 
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the time constraints on learning the tasks. As the needs changed, the military was among the first 
to use cross training to prepare individuals for doing more than one task. New technologies were 
often used first in military applications and then translated into civilian uses. 

► With the advent of new information based technologies and manufacturing processes and more 
attention to satisfying the specialized needs of customers, the basics requirements of the 
workplace are being altered. The results are multiple. Of particular note for this study is that job 
classifications are being collapsed, in some instances dramatically, from over six hundred to three 
or four in a single work site. Line workers, with more access to information, are expected to solve 
problems for customers, work in teams, adjust machinery, order supplies, and perform a host of 
other tasks that had previously been in the domain of indirect (or overhead) workers. 

These changes have led many representatives of business and industry to realize the importance of 
different types of skills, including the "softer skills" of communication, ability to work in teams, and 
problem solving. Carnevale, Gainer, and Meltzer (1990) coined the phrase the "New Workplace Basics" to 
capture the effects of the shifts currently underway in a wide range of employment settings in all parts of 
our society. 

► These new workplace basics have also led to calls for national education reform. These 
initiatives have taken many forms but one result is a growing recognition that it is neither efficient 
nor effective to have a chasm between the schools and the workplace. The recently completed 
work of the SCANS Commission helped to clarify that the new work-place basics must eventually 
become a part of the learning process for all students. It was not chartered, however, to delve 
into the needs of specific industries and the workers in them. Nor did the Commission have the 
responsibility to establish the on-going linkages between industries, education institutions, and 
workers that would assure the specificity needed to make industry and education based skill 
standards and competencies more pervasive in the United States. However, all of the work of that 
Commission recognizes the need for such linkages to exist. So, too, did the 1984 amendments to 
the primary federal vocational education legislation (Perkins I) recognize the need for improved 
linkages when states were encouraged to establish technical advisory committees of 
representatives from business and industry. 

The Commission on Work-Based Learning, originally established in 1991 by the Department of 
Labor, helped to build upon the work of SCANS and the legislative initiative contained in the Perkins II 
legislation (1990) to support collaboration between industry and education to develop skill standards 
systems. Developing common skill standards that can be used for various purposes by both education and 
industry was a core recommendation of this Commission. These mandates force the United States into 
largely uncharted waters, but provide some guidance in setting our direction. 
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While there has not been a pervasive practice of using industry to define the skill standards and 
competencies in this country, there exists a rich body of experiences from which to generate a national 
voluntary skills standards system. 

► The most ingrained standards and certification systems controlled by industry can be found in the 
professions such as medicine, law, architecture, and accounting. In these professions, the ability 
to practice is now intertwined with minimum postsecondary education requirements, attendance at 
a college that has been recognized by a special accreditation organization specifically established to 
assure industry input (e.g. schools of medicine), passage of tests established by the profession 
organizations, and in some cases requirements for the individual to take continual professional 
education courses. Though these systems are sometimes characterized as self-regulatory and 
voluntary in nature, they are now so intricately linked with State licensure requirements it is 
difficult not to consider the linkages mandatory and regulated. 

► A notable example of industry established skill standards are in the craft areas, particularly the 
construction trades, where movement of workers between a range of jobs is a necessity. The 
need for some form of "portable" proof that an individual was sufficiently trained to carry out the 
multiple tasks helped to bring employers and workers to the same table to develop skill standards 
systems. These industies have a rich and deep history of establishing skill standards. 

► Government regulation, or the threat of the same, has also influenced the action of industry 
representatives to come together to establish skill standards and develop working relationships 
with education institutions (often including the creation of special accreditation organizations). 

► In the grand American tradition of forming voluntary networks of peers there are thousands of 
national associations. Some of the 3200 industry based associations have become involved in 
establishing standards and competencies as a service to their members. Members are often 
desirous of having some form of personal recognition that distinguishes them from others for a 
variety of reasons (e.g. to give them a competitive edge, personal satisfaction, opportunity for 
increased remuneration). Meeting the needs of the members is a core function of industry based 
associations; however, the relative importance of any one association taking on the time 
consuming and somewhat costly task of developing and maintaining up to date industry skill 
standards and competencies is not a decision made lightly by most. Such decisions are often 
influenced by the maturity, size and type of industry, the size of firms within the industry, the 
range of commonly recognized occupations within the industry, and the ease of recruiting qualified 
individuals. Many of these associations efforts have focused the skill standards and certification 
systems on the needs of the managers, professional, and "highly" technical job classifications. 

This very brief overview has been provided to reflect just a few of the major historical and current 
issues that will affect the evolutionary process that is now being undertaken jointly by the Departments of 
Labor and Education to promote a voluntary network of industry based skill standards and competencies 
and certification systems. 
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This is a "first ever" undertaking; in order to develop a common framework that crosses the 
boundaries of the public and private sectors and considers multiple variations within both of those sectors, 
a set of ten organizational topics were developed to organize the materials. 

B. MAJOR TOPICS 

1. A Description of the Individual Program or Systems - This includes an overview of existing 
systems, and a description of the functional purposes for their development (e.g., individual 
certification and curriculum development). The target populations are identified and perquisites for 
access to the system (e.g., participation in training, work experience) are addressed. Courses of 
study, levels of certification, and management issues such as administrative authority and 
financing of the system are covered. 

2, The Extent of Coverage by the System(s) - This includes the range of industries and 
occupations covered by the existing systems, the overlap between industries and occupations, an 
estimation of the number of workers included, and how the coverage coincides with educational 
activity and government regulation of the occupation and industry. 

For these two topical areas every attempt has been made to be as comprehensive as possible, 
particularly for those occupations that have not traditionally required a four year college degree. This focus 
was selected by the federal government due to the continuing and mounting concerns articulated by 
businesses and industry representatives that a serious erosion of qualified entry level workers is permeating 
the U.S. labor force. 

In order to develop an in-depth understanding regarding how several of these systems (a 
combination of education driven and industry driven systems) have evolved, additional information is 
provided that includes the following topics: 



3. The Historical Development of the System(s) - This includes the driving forces or incentives; 
the barriers encountered; and what the time factors were for achieving the current levels of usage 
and coverage. 

4. The Partners' Roles and Responsibilities - Information is provided on the varying roles and 
reasons for involvement of the key partners, including educational systems and institutions, 
industry associations and employers, accrediting bodies, labor organizations, general government, 
apprenticeship programs, and other relevant bodies. 

5. The Standard Setting Processes - Information is provided on the different methods used in 
setting skill standards, who is involved, and the time, resources, and staff capabilities used in the 
process. 

6. The Occupational Analysis Processes - How occupations are classified and the variety of 
approaches used to determine the specific job requirements for a single occupation or common 
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grouping of occupations are discussed. The time, resources, and staff required for these 
processes are included where appropriate and feasible. 

7. The Assessment Processes - The procedures, instruments, processes, and types of anticipated 
outcomes of assessment vary and will be described. Specific information includes the types of 
testing and selection processes used, the relationships to instruction, the associated costs, 
methodologies used to assure consistency and integrity, methodologies used to insure against 
discrimination and other legal considerations, and approaches to maintaining and updating the 
assessment processes. 

8. The Instructional Approaches - Information on four major components of the instructional 
process are explored: the development of curriculum, the delivery of instruction, the selection and 
training of instructors, and the sites used for the delivery of instruction. 

9. The Processes of Maintaining and Updating Standards, et al - Here the focus is on what internal 
and external factors affect the maintenance and updating of standards and the time, resources and 
staff needed. 

10. How Information is Shared and Used In Skill Standards Systems - How, what, and with whom 
information is shared by various partners and external groups is described. Also included is 
information on how skills standards are used by employers in hiring and promotion practices; by 
educators/trainers in setting program outcomes and performance standards; in curriculum 
development and instruction; by government in establishing regulations and licensure policy; and 
by accrediting organizations. 

Two very intelligent men made conflicting observations: Mies Van der Rohe said that "God is in 
the details." Machiavelli is credited with having said that the devil is in the details. What follows are the 
details. 
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CHAPTER III 



JOB ANALYSIS 



A. BACKGROUND 

Jobs are critical because they are the building blocks of work organizations. They define roles; 
enable expectations, judgements, and evaluations; direct activities; form the basis of the reward system; 
and, through their fulfillment, result in the production of goods and services. Job analysis — the process of 
studying and coming to understand the various elements of a job - therefore, is critical to productivity in 
the United States. Moreover, job analysis is fundamental to an investigation of industry standards because 
it provides the "reality check," not only on what workers do as work, but also on the quality of their work 
performance. 

The purpose of this chapter is to investigate the issue and practice of job analysis. Consideration 
is given to the purposes and legal rationale for performing job analysis. Several alternative techniques for 
performing job analysis are described, and time factors, resources, and skills necessary for using the 
techniques are noted. Finally, job analysis as typically practiced in U.S. industry, education, and the 
military is discussed briefly, particularly in comparison to the formalized systems for performing job 
analysis. 

Job analysis is the systematic and analytic gathering, documenting, and analyzing ot information 
about actions employees take in performing the tasks incumbent to their jobs. That is, analysis deals with 
information about job content, and job requirements, as well as the context of the entire work organization. 
Therefore, job analysis is descriptive: it attempts to reduce to words the things that people do in human 
work, and it is both explanatory and predictive; it is always undertaken to meet a particular purpose or 
satisfy a specific organizational need. 



Among the purposes for which job analysis information has been used are the following: 

► Job Description - Includes identifying information about the purposes of the job as well and a 
summary of duties, responsibilities, activities, accountabilities and other specifications. 

► Job Evaluation and Classification - Includes the general process of allocating jobs in terms of 
their worth to an organization and in terms of their importance relative to each other, particularly 
for purposes of establishing salary pools and administrative hierarchies. 

► Job Performance/Performance Appraisal - Includes the systematic evaluation of employees 
working at their particular jobs to determine the relative efficiency and effectiveness of individuals. 
The information is used to make decisions about raises, promotions, and transfers. Also, it 
provides a formal way to acquire employee feedback. 
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►Training Design Includes identifying the necessary skills, knowledge and attitudes requisite to 
successful performance in a given job and translating that information into systematic instruction 
and learning opportunities. 

► Work Design - Includes the arrangement of work activities and tasks, so as to facilitate efficient 
rendering of services or production of goods. 

► Selection and Promotion Systems Includes the criteria and procedures used to screen, 
categorize, rank, promote, qualify, and select people for all jobs/positions within the organization. 

Recently, job analysis also has been used to help determine the fairness and legality of employment 
practices, a use that promises to grow with new legislation and increasing emphasis on the issue of pay 
and work equity/comparable worth. 

Historically, the terms "work analysis," "job analysis," and "occupational analysis" have been used 
almost interchangeably and have been discussed in management and training literature for most of the 
20th century. Taylor, in the early 1900's, referred to work analysis as one of the four great principles of 
scientific management. Later in the century, work analysis played a major role in the famous and/or 
infamous time and motion studies of the 1950s and 60s; it was fundamental to job training and 
performance appraisal systems of the U.S. military and, at least in part, responsible for their reputation as 
the most highly trained military force in the world. It played a critical role in creating wage and salary 
standards during the 1970s. 

More recently, job analysis has been used to plan and create training programs, to generate testing 
specifications for licensing and certification, and to formulate legal decisions regarding employment law. 
Over the next several years, these more recent uses promise to gain even greater attention as the nation 
addresses industry-wide training standards, broader and more comprehensive testing for licensing and 
certification, and new and more specific issues of employment-related discrimination. 



B. LEGAL ISSUES 



Legislation and the courts have had a significant impact on the Importance of job analysis since 
passage of the Civil Rights Act of 1964, as amended. That Act, which requires equality of employment 
opportunity without discrimination on the basis of race or sex, led to the production of the Uniform 
Guidelines that promulgate the responsibilities for carrying out the Act and assist employers to comply witl 
the law and regulations. 

The most recent of these Uniform Guidelines was created in 1978 and, in conjunction with the 
1 979 series of Questions and Answers regarding the Guidelines, created the required basis for use of job 
analysis in selection and promotion procedures. The Uniform Guidelines specifically state that when 
creating formal selection and promotion procedures, any method of job analysis may be used if it provides 
information required for the specific validation strategy that is employed. 
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Sparks (1979 and 1988) has pointed out the necessity of examining terms used within the 
Uniform Guidelines to understand the types of analysis allowed. Specifically, he argues that the Guidelines 
define job analysis as a detailed statement of work behaviors and other information relevant to the job. 
These behaviors involve observable and unobservable components and consist of the performance of one 
or more tasks. Knowledge, skills, and abilities are applied in the context of work behaviors with the 
delineation of what the worker does and requires that one utilize some form of observation to make that 
determination. 

Job analysis is critical to employment decisions because it is the foundation upon which 
employment tests, and inferences based on test scores, are built. Job analysis establishes validity. Three 
basic types of validity are presented as legitimate in the Guidelines : (a) content; (b) criterion-related; and 
(c) construct. Content validity is the degree to which some sample of tasks, questions, samples, or items 
on an assessment accurately represents the defined universe of possible content. Criterion-related validity 
is the degree to which the scores on a test accurately correspond to the performance, knowledge, and 
skills of those at work. Construct validity is the degree to which the scores on a test accurately measure a 
psychological characteristic of interest. 

The actual language of the Guidelines outlines three types of validation strategies and indicates 
how job analysis is important to each. For example, if the strategy is based on content validity, it must be 
demonstrated that the content of a selection procedure reflects and is representative of important aspects 
of performance on the job. With content validity, there should be a job analysis which: 
"includes an analysis of the important work behavior(s) required for successful 
performance and their relative importance and, if the behavior results in work product(s), 
an analysis of the work product(s). Any job analysis should focus on the work behavior(s) 
and the tasks associated with them. If work behaviors are not observable, the job analysis 
should identify and analyze those aspects of the behavior(s) that can be observed and the 
observed work products. The work behavior(s) selected for measurement should be 
critical work behavior(s) and/or important work behavior(s) constituting most of the job." 
If success in training programs is used in the selection process with content validity, one must 
establish the close relationship of training content with job performance. 

If the strategy is based on criterion-related validity, there must be a statistical relationship between 
scores on a selection procedure and the job performance of workers. The strategy must be based on a 
review of information about the job and should include a job analysis. However, certain criteria like 
production rate, error rate, absenteeism, and length of service may be used without a full job analysis if 
"the user can show the importance of the criteria to the particular employment context." If performance in 
training is used as a criterion, then success in training must be properly measured and the relevance of 
training to work must be demonstrated (a) through comparing training content and work processes or (b) 
through matching training and work performance measures. 
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If the strategy is based on construct validity, it must be demonstrated that the selection procedure 
measures a construct (something believed to be an underlying human trait or characteristic) and that the 
construct is important for successful job performance. There should be a job analysis for construct validity 
that shows (a) the work behaviors required for successful performance of the job, (b) the most critical or 
important work behaviors in the job, and (c) identification of the construct(s) believed to underlie 
successful performance of these critical or important work behaviors. Then the validity of the relationship 
of the selection procedure to the construct and of the construct to work behavior must be demonstrated 
empirically. 

In addition to these requirements established by the Uniform Guidelines, case law from the U.S. 
Supreme Court also has helped establish the need for and importance of job analysis for making 
employment decisions. Two cases in particular, Griggs v. Duke Power (1971) and Albermarle v. Moody 
(1975) have helped establish requirements. More specifically, in Griggs v. Duke Power the Supreme Court 
established the principle of job relatedness, although they did not specifically mention job analysis by name. 
The court ruled that employment practices that were not job related were prohibited. Specifically, they 
said that Title VII of the Civil Rights Act of 1964, as amended, requires evidence of validity in a selection 
procedure if that selection procedure adversely impacts the selection of minority, group employees, or job 
candidates. In Albermarle v. Moody the court reinforced the importance of job analysis by holding that any 
validation effort of selection processes that did not include job analysis was insufficient. 

After these initial cases established the need for performing and using job analysis information in 
creating procedures for making employment selection and promotion decisions, more recent court cases 
have begun to establish procedures that must be followed when performing a job analysis. Thompson and 
Thompson (1982) and Kuehn, Stallings, and Holland (1990) reviewed dozens of state and federal cases 
and determined that a number of principles derived from this case law prescribe and proscribe both the 
practice of job analysis and the practice of testing. The principles that relate to job analysis include the 
following: 

1 . "In constructing an examination that will have "content" validity, the preferred course is to 
have an empirical analysis made of the position for which it is given, usually by experts or 
professionals in the field. Such an analysis requires a study be made of the job, of the 
performance by those already occupying it, and of the elements, aspects and characteristics that 
make for successfully performance." ( Chance v. Board of Examiners, 1971). 

2. "... survey the relative importance of the various skills involved in the job in question and the 
degree of competency required in regard to each skill. It is conducted by interviewing workers, 
supervisors, and administrators; consulting training materials; and closely observing actual job 
performance." ( Vulcan Society v. Civil Service Commission, 1973.) 

3. "The cornerstone in the construction of a content valid examination is the job analysis. 
Without such an analyses to single out the critical knowledge, skills, and abilities required by the 
job, their importance relative to each other, and the level of proficiency demanded to each 
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attribute, a test constructor is aiming in the dark ..." (Kirkland v. Department of Correctional 
Services , 1974). 

4. "A task-oriented analysis was not done during the development of the 1975 trooper 
examination. To the extent that this was not done, the development of the 1975 trooper 
examination based on a content validity strategy was not in accordance with professional 
standards or in compliance with federal guidelines on employee selection." ( U.S. v. State of New 
York , 1979) 

5. "Job incumbents are job knowledge specialists and should be part of the job analysis." 
( Gillespie v. The State of Wisconsin, 1985) 

6. "Regional or job context variability must be taken into account in test content domain 
definition. Incumbent sample must be representative." ( Burny v. The City of Pawtucket , 1983; 
Allen v. Isaac , 1986). The Civil Rights Acts of 1964, as amended, the Uniform Guidelines 
together with their Questions and Answers , and case law rendered by State and Federal Courts 
establish both the need for performing job/occupational analysis as a basis for constructing 
procedures for making employment decisions and offer suggestions about how job analyses must 
be performed in order to ensure that the results of a particular job analysis are valid. 



C. JOB ANALYSIS TECHNIQUES 



Job Analysis Techniques are the formal procedures that researchers follow in order to obtain and 
analyze information about the specific jobs. Cornelius (1988) analyzed a number of job analysis studies 
and determined that the components of the job analysis process fall into four broad categories. Those 
categories, as presented in Table 3: Components of the Job Analysis Process, are sources of job data; 
methods of obtaining job data; type of job descriptor; and purpose of the analysis. 



TABLE #3 

COMPONENTS OF THE JOB ANALYSIS PROCESS* 



Sources of Job Data 

Job incumbents 

Supervisors/managers 

Clients 

Training & Evaluation Dept's personnel 

Observers/analysts 

Job materials/industry information 

Training materials 

Published information and other data sources 



Methods for Obtaining Job Data 
Observation 
Questionnaires 
Work diaries/logs 
Individual interviews 
Group interviews 

Reading/analysis of printed and/or published materials 
Structured group process 
Job expert conferences 
Job performance by analyst 
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TABLE #3 

COMPONENTS OF THE JOB ANALYSIS PROCESS* (cont.) 
Type of Job Descriptor Purpose of Analysis 



Tasks and activities performed 

Worker oriented processes 

Abitities and aptitudes 

Skills, knowledge and attitudes 

Products, outcomes, and success criteria 

Responsibilities 

Physical/psychomotor and psychological demands 
Education and training requirements 
Contextual components and interaction 
Decisions and their components 



Job specifications and prerequisites 
Job descriptions 

Selection systems, testing, licensing, and certification 

Recruiting strategies 

Promotion systems 

Wages and salary administration 

Job classification/evaluation 

Work design/redesign 

Planning, materials development and delivery of training 
Performance appraisals 

Planning for work force numbers and skills for specific jobs 
Equal opportunity considerations 



*Adapted and expanded from, E.T. Cornelius III. "Practical Findings from Job Analysis Research," in The 
Job Analyses Handbook for Business, Industry and Government, ed. Sidney Gael, 1988. 

As indicated in Table 3, a number of purposes can be fulfilled through the use of job analysis 
information; the particular concerns of this paper are selection and promotion procedures. The Uniform 
Guidelines and case law have indicated that the most important types of job descriptors available through 
the job analysis process are tasks, activities, skills, knowledge, and attributes. However, this issue is still 
open to debate, particularly as it relates to the potential discriminatory practices of assessment processes. 
More specific information about these types of descriptors will be presented in another chapter of this 
report, the one dealing on assessment techniques. The two remaining components pointed out by 
Cornelius, sources of job data and methods of obtaining job data, are the particular concerns of this 
chapter. 

The sources of job data include all the factors that are identified in Table 3 including, for example, 
job incumbents themselves as well as their supervisors, subordinates, and clients; training department 
personnel; observers; and job materials, training materials, and additional published information. The 
methods for obtaining the data include observation, questionnaires, individual and group interviews, and 
analysis by experts. 

Since the 1950's a number of specific job analysis techniques have been devised by researchers as 
systematic plans for generating useful job analysis information. Among the dozens of techniques that have 
been created are such notable job analysis procedures as the Critical Incident Technique, the Functional 
Job Analysis, the Position Analysis Questionnaire, DACUM (Designing A CurriculUM), formal observation, 
Job Element Analysis, Task Inventory Analysis and its various derivatives, and other similar techniques. 
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While it is beyond the scope of this project to investigate each of these various techniques for performing 
job analysis, some brief comparisons seem appropriate. 

Functional Job Analysis (FJA) 

One of the more notable and widely used techniques for performing job analysis is the Functional 
Job Analysis (FJA) developed by Sydney Fine as part of the activity to produce the third edition of the 
Dictionary of Occupational Titles. The functional job analysis is a comprehensive approach that 
investigates the interactions among work, workers, and work organization through a multi-dimensional 
information collection activity. Major components of the FJA are: 

1. Purpose, goals and objectives -- identified by key actors within the organization. 

2. Identify and describe tasks — Articulating specifically what the worker does and what gets 
done. The tasks are described as actions or action sequences, grouped through time so that they 
contribute to a specific accomplishment or product. Eventually tasks are written according to a 
standardized format by the job analyst. 

3. Task analysis - Considering what a worker does according to seven scales: the data scale, the 
people scale, the things scale, the worker instructions scale, the reasoning scale, the mathematic 
scale and the language scale. 

4. Performance standards -- Defining criteria for evaluating the results of task performance. 

5. Training content - Specifying information on the functional content specific arnd adaptive 
requirements of the job. 

The FJA information is developed by trained analysts who interview workers and supervisors and 
who observe work. For example, the job analyst first interviews upper level management to determine 
organization mission, purpose, goals and objectives so that there is a context into which to place task 
analysis. Next, the analyst undertakes a series of steps to produce a task bank: a review of existing job 
information; arrangement of a 1-2 day work session of experts on the job in question (incumbents in the 
occupation under consideration); welcome subject matter experts; demonstrate how FJA task statements 
work; list outputs/products/services of jobs; generate tasks; refine task bank; generate performance 
standards; edit task bank; and administer final task bank inventory. 

It has been used extensively throughout industry in managerial, professional, and craft situations. 
The FJA has been used to classify jobs (within the U.S. Employment Service), to generate work sample 
tests (for the international Union of Operating Engineers), and to develop performance standards for work 
(for the Department of Housing and Urban Development). The output also has been used to develop 
training materials and for job evaluation. 

The FJA uses multiple methods to collect useful information. It is relatively simple to use, but 
requires a job analyst's time and requires specific training for the analyst. The typical FJA might require 
five days of data collection per occupation, plus preparation and training time; in addition, another several 
days usually are required to rewrite the task bank and achieve final consensus among subject matter 
experts. Time is also required to apply the information to the specific purposes for which the FJA was 
conducted in the first place. 
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Critical Incident Technique (CIT) 

A second frequently used job analysis technique is the Critical Incident Technique (CIT), developed 
by John Flanders in the early 1950s. The technique involves several basic steps. First, the analyst 
identifies critical incidents, incidents that illustrate behaviors that are effective or ineffective when 
accomplishing the aims of the job. Incidents are collected from supervisors, job incumbents and anyone 
else who may be in a position to evaluate a particular job. There must be agreement among the 
respondents about the aims or purposes of the job. Information can be collected through individual and 
group interviews as well as through observations or structured questionnaires. 

While an unstructured approach may be used to recall and/or record critical incidents, usually a 
structured recording format will improve the time efficiency of the session and the quality of the output. In 
particular, prompting questions such as, "What are the circumstances leading to the example?"; "What 
was done?" and "What were the consequences of the action?" usually will help the process. 

The second step is to classify the critical incidents into categories of behavior. Classification 
schemes typically are developed by a trained analyst and result in a descriptive format for the different 
kinds of activities performed at work. Before the incidents can be categorized, the items must be checked 
, for with the job incumbents who provided them (to ensure accuracy), they must be edited to conform with 
the "rules for well-written incidents," and are usually shortened to about 40 words. Next, the job analyst 
must sort the incidents into groups or categories. This process can be performed by working with job 
experts or existing printed materials to devise broad categories before reading the incidents or it can be 
completed by applying statistical techniques to task ratings. Usually, the categories will be based on the 
purposes of the CIT analysis. 

The final step in the CIT is the retranslation of the incidents. That is, the analyst prepares a survey 
of incidents for 10-20 raters to judge on a multi-point scale from ineffective to excellent. Thus, ineffective 
or poorly written incidents can be revised before use. 

The Critical Incident Technique has been used to design jobs and develop training curriculum in a 
variety of occupational settings. These were the purposes for which the CIT was designed. Typically, 
however, it has been used as a supplemental data collection technique for other purposes such as job 
selection system design. 

The CIT is time consuming, requires someone with special training, and produces data that are not 
necessarily representative of the entire range of tasks performed in the occupation. The initial data 
collection takes from several hours to several days, depending on which data collection technique is used; 
the editing and dimension-defining techniques require several days of an analyst's time; and the 
retranslation verification requires several days of an analyst's time and a half-day of 10-20 raters. 

Position Analysts Questionnaire (PAQ) 

A third major type of job analysis technique is the Position Analysis Questionnaire (PAQ) and 
techniques similar to it. The PAQ was originally developed by Ernest McCormack & Associates at Purdue 
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University as a strategy to enable job analysts and Industrial psychologists to better define, quantify, and 
compare job characteristics. The PAQ questionnaire uses almost 200 elements of job activity that are 
organized into six broad categories: 

1. Information input or how and where the worker gets information; 

2. Mental processes or the reasoning, decision-making, and planning activities that the employee 
uses; 

3. Work output or the activities that a worker performs as well as the tools used in that 
performance; 

4. The relationship with other people, or who else is involved in performing the job; 

5. The job context or the physical and social parameters including things like noise and obligations 
that a job incumbent may encounter; and 

6. Other job characteristics which pick up issues like irregular work schedules or repetitive 
activities. 

Items are rated on scales such as amount of time or importance to the job, and analyzed by job 
analysts who typically complete the questionnaire and group information into categories. 

The PAQ allows derivation of job dimension scores for occupations as determined through factor 
analyses of several thousand jobs. The dimensions provide a kind of "structure* to human work and 
represent broad behaviors that are required in some fashion or another in most jobs. Each of the six 
divisions includes its own set of dimensions. For example, the Work Output Division includes the 
dimensions of using machines, tools, and equipment; general body versus sedentary activity; control and 
related physical coordination; skilled/technical activity; controlled manual/related activity; use of 
miscellaneous equipment/devices; handling, manipulating related activity; and physical coordination. 

In order to use the PAQ, several operations must be performed. The reliability of PAQ data must 
be established by computing correlation coefficients between multiple raters and between initial and later 
rating. In addition, PAQ data must be edited to ensure its completeness and comparability with the 
requirements of the computerized program that will be used to process it. Moreover, analysts with 
information about the specific industry and the jobs in question must be trained in the PAQ and the 
procedures for a PAQ analysis. A PAQ Job Analysis Manual is available to assist with the process. Still, 
training and support can take a week; and the level j2ff training has a direct bearing on how lo j the actual 
analysis will take/cost. 

The position analysis questionnaire has been used to create compensation programs and perform 
job evaluation particularly in finance, health, manufacturing and service industries. Because the PAQ is a 
combination of contextual information, content, and work requirements, it is sometimes difficult to 
determine how the items relate to each ether or how to use the information to create training programs 
that are explicitly focused on specific tasks. Nevertheless, it has been used widely and has spawned a 
number of simila r techniques. 
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Developing A Curriculum (DACUM) 

DACUM is a newer job analysis technique, originally devised in Canada and popularized by 
vocational education in the United States. The DACUM process was refined at the Ohio State University 
Center on Education and Training for Employment under the direction of Dr. Robert Norton. DACUM charts 
are used in curriculum development, worker training programs, test development, and needs assessment. 
The DACUM process involves role incumbents and supervisors in small group interview sessions where 
they work together to identify job tasks processes and order those tasks and processes temporally. 

DACUM sessions usually involve 8-15 workers and supervisors who work for half a day to several 
days with a trained facilitator to generate the information. First, broad duties are identified. Then, these 
duties typically are listed on sheets of paper or cards on a board. The facilitator asks the group first to 
arrange the duties and then to identify the tasks performed by the job occupant for each broad category of 
activity. These tasks are in turn analyzed for requisite knowledge and skills and then collapsed into main 
tasks and rated on a scale for frequency and for importance. 

The result is a listing of tasks and activities for any particular job, as developed by job incumbents 
and supervisors that can be used for developing training materials that are highly task specific. 

DACUM functions somewhat like an abbreviated version of FJA. It is not especially inexpensive; 
however, it requires a facilitator and 8-15 job incumbents who miss work. Further, the facilitator must be 
familiar with the technique and trained in group process management. 

According to Dr. Norton, DACUM operates on three basic assumptions: 

1 . Expert workers can describe and define their job more accurately than anyone else. 

2. An effective way to describe a job is to define the tasks expert workers perform. 

3. Ail tasks demand certain knowledge, skills, tools, and attitudes in order to be performed 
correctly. 



Procedurally, DACUM follows a sequential order as follows: 

1 . Orient the committee 

2. Review the occupation 

a. Conduct brainstorming 

b. Review the organization chart 

3. Identify general areas of responsibility (duties) 

4. Identify specific tasks performed 

5. Refine/review task and duty statements 

6. Arrange the task and duty statements 

7. List requirements: 

a. General knowledge and skill requirements of the job 

b. Desirable worker behavior attitudes and traits 

c. Tools, equipment, supplies, and materials 

8. Other desired options 
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A task must be defined in very specific performance terms. This is often difficult to achieve. 
Consider the tasks required to solder parts together. The statement, "Understand how to solder," would 
not be considered a proper task statement since it does not define a specific task. A better statement 
would be, "Solder resistors to a PC board." Even this statement could be broken down into smaller parts 
until the statement defines only a single task. The skilled facilitator establishes the degree to which the 
tasks must be defined. Then, through a process of negotiation, clear, sufficiently detailed performance 
objectives are established. These objectives must leave no doubt as to the limits of the task. Action verbs 
used in the descriptions leave no doubt as to the specific task. The tasks define specific jobs under the 
broader "Competency Area." 

According to Norton (1992), a secondary product of DACUM for schools is public relations with 
business and industry. DACUM often produces long-term results, as indicated in the following reports by 
schools. 

1 . Offers of equipment and supplies 

2. Offers of teaching personnel in emerging areas 

3. Requests for in-service training to meet local needs 

4. Increased enrollment in adult upgrading programs 

5. Increased support of educational institutions by local business, industry, labor, and 
management 

Job Task inventory (CODAP and the Extended Search) 

Yet another job analysis method is the computer analyzed job-task inventory approach. This 
approach, often called CODAP, was originally developed by Dr, Raymond Crystal and others for the U.S. 
Air Force. The CODAP procedure is particularly useful in working with large populations in diverse 
locations because, as a survey, it allows for collapsing large amounts of data into categories, and 
manipulating the data statistically to look for relationships and generate descriptive and prescriptive 
information. 

The process involves two basic components. First, a task inventory is developed and used to 
collect the data about hundreds of tasks from a variety of workers. Data are collected on specific tasks, 



the time spent on various tasks, tools used, and the general worker attitude toward the job. Next, the' data 
are fed into a interactive computer program to analyze, organize, and report the information from the task 
inventory. The computer program is called the Comprehensive Occupational Data Analysis Program or 
CODAP, hence the name. Over 50 standard programs are available for data analysis, covering basic 
processes such as data preparation and validation, selection of individuals into groups, computation of 
summary information, comparison of summary information, and prediction. 

The most time consuming portion of the CODAP procedure is the initial construction of the task 
inventory questionnaire. Typically, it is created by job analysts who review all available information on the 
type of job and who interview and observe workers and supervisors to generate and edit the list. Available 
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information includes library data bases, instructional materials, industry information, and other related task 
lists. The process of reviewing all such information now is called an "extended search" and sometimes is 
used as a job analysis technique by itself. 

The CODAP survey form usually consists of several types of questions. First, there usually is a 
general "background information" section on which the respondent indicates data like name, job, 
educational level, career aspirations, and so forth. Often the background section contains checklist 
questions that ask respondents to indicate things like tools and equipment used previous to current job, 
work attitudes, prior training, and so forth. Usually the background information is used to develop groups 
or categories of respondents for analysis purposes. 

The second major portion of the inventory is the task list itself. The inventory usually lists all 
tasks, grouped under broad duty headings. Each task is stated as an activity, beginning with a 
performance verb. Incumbent workers fill out the inventory indicating if they perform a task and 
suggesting the relative amount of time they spend doing each task relative to other tasks. Supervisors 
provide additional data about tasks, such as the length of the required learning curve before an employee 
completes the task successfully. 

After it is collected, data are processed, analyzed, and reported through use of the available 
computer programs. The result is job content information such as a description of the work that is 
performed. Further, the data can be analyzed against the background data of groups of respondents as a 
way to analyze trends. 

A disadvantage of the job-task inventory is that it really does not identify requirements, skills, 
knowledge, or other characteristics necessary to deal with a particular job except through self-identification 
by respondents if the items have been included in the background information section. 

The Job-Task Inventory Analysis process has been used for a number of purposes including 
developing job descriptions, classifying and evaluating jobs, and in conjunction with other data, for 
establishing evaluation systems. However, the most frequent use is to plan and/or adjust training programs 
so they more accurately reflect the actual practice of work. 



Another technique for performing job analysis is direct observation of work. Observation has a 
distinct advantage in providing depth of analysis, validity, and flexibility of purpose. However, it is time 
consuming, difficult to analyze, and susceptible to subjectivity among observers and the quality of their 
training. 

Most often, observation involves some period of structured observation where tasks and activities 
are noted in pre-designed categories, particularly categories related to non-verbal, spatial, linguistic, and 
psycho-motor activity. In addition, usually there is a need for periods of unstructured observation where 
the observer notes and considers context and the interrelated patterns of behavior in the workplace. In 
both situations, field notes are required that must be transformed into coded data and analyzed to produce 
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some formal written description of the activity in question. Frequently, observation has been used to 
develop job analysis of occupations where psycho-motor activities hold equal sway with intellectual 
activities and where contextual situations are particularly important. 

Several additional comments are necessary regarding observation as a technique. First, it is almost 
the only way to get a good representation of the organizational context in which work occurs. Norms, 
rules, "informal" structure, interaction patterns, and "short cuts" are observable, but available in almost no 
other way to a job analyst. However, other information such as the relative frequencies of task 
performance or the importance of various tasks are very difficult to determine through small numbers of 
observations. 

Observation takes time - time to prepare and arrange, time to observe, and time to transcribe and 
interpret what was observed. And often the transcription and interpretation process can take twice as long 
as arranging for and conducting the observation. In general, the time required to conduct observational 
studies is much greater than studies using interviews or questionnaires, given the same size sample. 
However, as one of the series of data gathering techniques, it provides very useful information and usually 
can serve more than one purpose at the same time. 

Observation requires trained observers. Usually training will require one or more days during which 
time the project is discussed; the observation and coding system is explained; practice observations are 
performed; and feedback regarding observation errors is addressed. Practice must be emphasized as a way 
to reduce inter-rater differences. Training ceases when the inter-rater error rate becomes negligible and 
when everyone understands the potential problem areas of "halo effect" loss of detail, contrast errors, and 
the like. Sometimes technological aids such as audio procedures, videotape, and the like are used in 
training and in actual observation as a method to overcome potential errors. 

One of the most problematic concerns of observation is determining how many observations to 
make. If observation is used to supplement other information, then many fewer observations are required. 
Further, in the context of content validity, if one is confident that the observations are representative of the 
population of possible observations, the number of observations may be few but thorough. However, with 
construct and criterion-related validity where statistical measures are expected, the number of observations 
may be more important. For those situations, the technique of activity sampling - specific numbers of 
observations for short periods of time -- is useful. 

Activity sampling usually involves performing a given number of random, short term observations 
of job incumbents to note task activity, frequency, and the amount of time spent performing such task. 
Frequently, activity sampling is used to estimate the proportion of a worker's time spent performing any of 
the several categories of activity, incumbent to the job. The number of observations required is based on 
how precise the researcher wants to be regarding worker time use. The more precise the desire the higher 
the number of required observations. Pape (1988) has developed a somewhat complicated formula for 
determining how many observations one must make to achieve given levels of confidence and precision. 
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THE MOST EFFECTIVE APPROACHES 



The question then becomes, "Which of the variety of job analysis techniques is most effective?" In 
spite of the fact that a number of studies have been performed around that question, no clear conclusions 
can be drawn. Several studies suggest that both simple or complicated analysis methods frequently 
produce adequate data (Hogan & Fleishman, 1979; Hogan and others, 1980; Jones and others, 1982). 
Other studies document that supervisors and incumbents agree about the identity of tasks they perform 
but disagree about the ratings (levels) of the attributes that are required (O'Reilly, 1973; Smith & Lakel, 

1979) . Still other studies demonstrate that trained observers give fairly similar results concerning job 
characteristics to job incumbents when both are compared on the same techniques (Jenkins and others, 
1975) even though supervisors and subordinates may attach different meanings to the dimension of work. 
That is, they suggest a different level of importance and frequency to task dimensions and possibly would 
organize work differently. And yet in another set of studies (Arvey & Begalla, 1975; Cornelius & Lyness, 

1980) , researchers found that when incumbents and supervisors were used as the source of information, 
the better educated respondents provided the more accurate data, thus suggesting that even though job 
tenure and performance level were relatively unimportant in choosing incumbents or respondents, the 
educational level was important. 

In another set of studies, Hermann, Willett and Rayner (1988, 1989) have investigated the types 
of analysis strategies that work best for producing particular types of job analysis data. Hermann and his 
colleagues argue that several types of job analysis data are produced during the studies. Not only are task 
descriptions produced, but the studies generate suggestions of skills and knowledge competency as well as 
information about the importance and extent of performance for any given task or competency. 

When these researchers investigated the Critical Incident Technique, DACUM, the Extended Search 
of Information (the library search of all published data on the job the first step of the CODAP Job Task 
Inventory) and Observation techniques for providing each of these three types of data, they determined 
that while each technique produced useful task descriptions, DACUM and the Extended Search/Job Task 
Inventory were considerably more effective and efficient than were Observation and the Critical Incident 
Method. More specifically, DACUM and the Extended Search independently generated about 73% of the 
competencies on a master list of competencies provided through compiling results of all of these 
techniques. However, when DACUM and the Extended Search were combined, they produced 94% of the 
total competencies that were identified. Therefore, Hermann and his associates recommend the only truly 
effective way to conduct job analysis is through using multiple methods to deal with the entire range of 
tasks and competencies that might be produced in a job analysis. 

Three additional points need to be made about job analysis techniques. First, a technique for 
collecting broad-based information from groups of individuals -- Focus Groups — has gained recent 
popularity but probably is not a legally or technically acceptable method for collecting job information. It 
lacks structure and rigor even though it can be representative. 
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Second, whatever methods are used, they must involve job incumbents in producing information 
and must include information that incorporates regional differences from across the United States. 

Third, the U.S. Department of Labor, through sponsoring the work of the SCANS commission, has 
helped focus the attention of many interested parties on the importance of skills and competencies that are 
part of the job. The SCANS effort, through the testimony and work of academic, government and business 
officials, developed expressions of the basic skills necessary both for success and for meeting the needs of 
most "high performance" workplaces. The ideas were refined and extended through conducting over 600 
community meetings to discuss the initial findings. Further, the Commission has provided thousands of 
copies of their work products as technical assistance to local areas and projects that want to move to 
*dopt the SCANS model. 

The SCANS model is a type of standards language that provides categories of functional activity 
that occur, to some extent, in all jobs. The Commission developed a two-part rubric that includes (a) 
foundation skills (basic sills, thinking skills, and personal qualities) and (b) competencies (resources, 
interpersonal, information systems, and technology). Figure 1 , SCANS Analysis displays findings of the 
Commission. Clearly the work advanced the ideas of a job task analysis by emphasizing important and 
heretofore ignored critical areas of work. 

D. JOB ANALYSIS IN PRACTICE 

Job analysis in practice comes in a variety of scales, detail, and cost. Further, different institutions 
within the U.S. economy may approach the process in somewhat different ways. To demonstrate those 
differences, it is necessary to look only at examples from three different "sectors" of our society - 
business, education and the military. 

Industry 

Within American industry, job analysis has been used extensively to deal with job classification 
and, in more recent years, has been used to help determine training input. Apprenticeship programs have 
used task analysis for several hundred years to establish training input and set evaluation criteria for 
determining individual advancement. However, when the type of job analysis actually performed in 
American industry is examined it becomes clear that only occasionally is a single system used, either 
entirely, systematically, or to the exclusion of other techniques. Also, the gathering of job analysis data 
often has unanticipated positive results. One such instance occurred when the International Union of 
Operating Engineers used the Functional Job Analyses to collect information to deal with discrimination 
allegations. In the course of performing the analysis, they developed considerable information that has 
since been used to create and redesign their extensive training program. 

Other occupations also have undertaken systematic analyses of specific occupations within their 
industries. For example, clerical jobs have been analyzed through job/task inventory analyses applied with 
the CODAP data processing program. Jobs in finance, banking, sales, and management have been 
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analyzed using the position analysis questionnaire. The job of police officer has been investigated a 
number of times, usually by employing a combination of the critical incident method and some kind of task 
analysis inventory. 

The combination of techniques seems to be gaining favor, especially because it allows not only for 
improved validity but also for the data to be put to multiple purposes. In blue collar construction trades, for 
example, often a task analysis inventory in combination with group interview techniques and formal 
observation seems to be the preferred methods of collecting information. 

Recently, the Laborer's - AGC, the Joint Labor Management arm of the Laborer's International 
Union of North America and the Associated General Contractors of North America has undertaken 
occupational analysis of several components of the work that skilled construction laborers historically 
perform. In that analysis, the laborers used the following combination of methods in order to determine 
their job tasks and to work from those job tasks to identify skills, knowledge, and attitude competencies. 
Their activities illustrate the current practice of job analysis in U.S. industries. Among their activities were 
the following: 

► They reviewed all existing, printed material about the jobs, both within their organizations and 
from other organizations that were associated with the particular craft. For cement finisher, for 
example, that included reviewing materials produced in Canada regarding the task analysis of the 
trade, materials produced on new techniques by the World of Concrete as well as specific 
manufacturers, the skill certification requirements of the American Concrete Institute, and the 
published training and testing information available in written and computerized forms. 

► They convened a nationally representative sample of 150 training directors, supervisors, and 
tradesmen to work together to review and suggest tasks, and to order tasks according to time, 
importance, and frequency. 

► They interviewed more than 70 tradesmen and supervisors in group meetings to determine 
specific critical incidents and to prioritize tasks that were important to effective and ineffective 
work in the trade. 

► They prepared a task inventory based on all this information that they sent to a large 
representative sample of supervisors and tradesmen to have them respond to the validity, 
importance, and frequency of the task and to ask them to add additional tasks and competencies 
that might be important. 

► They employed trained/experienced observers to frequent the job to verify and clarify the 
information that was produced. 

Based on this information, they are developing a composite description of tasks, skills, 
competencies, and criteria of performance necessary to operate effectively in that given portion of their 
craft. The information is being used both to upgrade their training program and to build a series of work 
sample performance tests to certify the skills of their tradesmen. 
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FIGURE 1: SCANS ANALYSIS 



A THREE-PART FOUNDATION 
Basic Skills: Reads, writes, performs arithmetic and mathematical operations, listens and speaks 

A. Reading - locates, understands, and interprets written information in prose and in documents such as manuals, graphs, and 
schedules 

B. Writing - communications thoughts, ideas, information, and messages in writing; and creates documents such as letters, 
directions, manuals, reports, graphs, and flow charts 

C. Arithmetic/Mathematics - performs basic computations and approaches practical problems by choosing appropriately from a 
variety of mathematical techniques 

D. Listening - receives, attends to, interprets, and responds to verbal messages and other cues 

E. Speaking - organizes ideas and communicates orally 

Thinking Skills: Thinks creatively, makes decisions, solves problems, visualizes, knows how to leam, and reasons 

A. Creative Thinking - generates new ideas 

B. Decision Making - specifies goals and constraints, generates alternatives, considers risks, and evaluates and chooses best 
alternative 

C. Problem Solving - recognizes problems and devises and implements plan of action 

D. Seeing Things in the Mind's Eye - organizes, and processes symbols, pictures, graphs, objects, and other information 

E. Knowing How to Learn - uses efficient learning techniques to acquire and apply new knowledge and skills 

F. Reasoning - discovers a rule or principle underlying the relationship between two or more objects and applies it when solving 



Personal Qualities: Displays responsibility, self-esteem, sociability, self-management, and integrity and honesty 

A. Responsibility - exerts a high level of effort and perseveres towards goal attainment 

B. Self-Esteem - believes in own self-worth and maintains a positive view of self 

C. Sociability - demonstrates understanding, friendliness, adaptability, empathy, and politeness in group settings 

D. Self Management - assesses self accurately, sets personal goals, monitors progress, and exhibits self-control 

E. Integrity/Honesty - chooses ethical courses of action 

FIVE COMPETENCIES 
Resources: Identifies, organizes, plans, and allocates resources 

A. Time - Selects goal-relevant activities, ranks them, allocates time, and prepares and follows schedules 

B. Money - Uses or prepares budgets, makes forecasts, keep records, and makes adjustments to meet objectives 

C. Material and Facilities - Acquires, stores, allocates, and uses materials or space efficiently 

D. Human Resources - Assesses skills and distributes work accordingly, evaluates performance and provides feedback 

Interpersonal: Works with others 

A. Participates as Member of a Team - contributes to group effort 

B. Teaches Others New Skills 

C. Serves Clients /Customers - works to satisfy customers' expectations 

D. Exercises Leadership - communicates ideas to justify position, persuades and convinces others, responsibly challenges 
existing procedures and policies 

E. Negotiates - works toward agreements involving exchange of resources, resolves divergent interests 

F. Works with Diversity - works well with men and women from diverse backgrounds 

Information: Acquires and uses information 

A. Acquires and Evaluates Information 

B. Organizes and Maintains Information 

C. Interprets and Communicates Information 

D. Uses Computers to Process Information 

Systems: Understands complex inter-relationships 

A. Understands Systems - knows how social, organizational , and technological systems work and operates effectively with them 

B. Monitors and Corrects Performance - distinguishes trends, predicts impacts on system operations, diagnoses deviations in 
systems' performance and corrects malfunctions 

C . Improves or Designs Systems - suggests modifications to existing systems and develops new or alternative systems to improve 



Technology: Works with a variety of technologies 

A. Selects Technology - chooses procedures, tools, or equipment including computers and related technologies 

B. Applies Technology to Task - Understands overall .ntent and proper procedures for setup and operation of equipment 

C. Maintains and Troubleshoots Equipment - Prevents, identifies, or solves problems with equipment, including computers and 



a problem 



performance 



other technologies 
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Military Services 

The military services have been involved in occupational analysis since World War II. Their process 
has evolved from the review of a simple position description to a systematic analysis approach, involving 
complex analysis, design, development, implementation and evaluation methods. All branches of the 
armed services provide an abundance of information related to the development and refinement of 
occupational analysis processes. Each service, however, has a slightly different process for addressing 
each area of systematic analysis. 

The military job families or clusters are classified into the Military Occupational Specialties 
(M.O.S.). They examine the total number of positions and tasks related to each arm of the military service. 
The Navy and Marines identify 262 officer and 474 enlisted categories of jobs. The Air Force identifies 
200-300 occupations for enlisted personnel, and the Army identifies 350 enlisted personnel occupations, 
of which 179 directly correlate with similar civilian jobs. The Air Force indicated that in June, 1991, an 
"Occupational Conversion Menu" was published to translate job skills from military positions to those of a 
similar nature in civilian life. Conversion tables were established for the purpose of assisting personnel 
exiting military service in matching the skills they acquired while in the service to those required in similar 
civilian jobs. The Army, along with Ohio State University, has developed studies to examine the transfer of 
Army "alumni" to civilian jobs. Their studies show that 50% of those who left the Army with skill training 
in a transferable occupation found employment at a higher rate of pay than others, according to John 
Buckley, U.S. Army. 

Although similarities may be found in the systems analysis processes, many distinctions are found 
in the specific jobs analyzed and the environment in which tasks may be performed. All of the services 
indicated that they perform 100% surveys of personnel when they are doing occupational analysis of a 
field. As expected in a closed system, the returns on the survey are quite high. The Air Force notes that 
they receive around 95% return on their surveys. The other 5% may be unavailable due to combat duty, 
etc. Other services also indicate a top return on the surveys. 

Since the objective of all military services is to at all times have a currently skilled work force, 
prepared to meet specific standards, each group updates their occupational analysis of functional groups on 
a regular basis. The Army updates their occupational skills at least every two to three years. Some 
occupations are updated annually. The Air Force updates their occupations at least every four to five 
years. Again, occupations which have a rapid technology change or method of work change are evaluated 
as frequently as needed. The Navy and Marines indicate that they update their fields on a two to three 
year cycle, except for the new technical fields. 

The classification system starts with functional groups which relate to 37 occupational fields. All 
military services update their occupational analysis of a functional group at least every two to three years. 
In the new technical fields, the update is performed more often. 

During the late 1960s and early 1970s, the Air Force developed an analysis system which 
addressed a more in depth process of classifying job types. 
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The current Air Force Occupational Analysis Program (Tarteii) includes a closed-loop process 

which: 

1. Establishes an inventory of job skills built through the interview process; 

2. Develops a statistically valid survey; 

3. Conducts surveys; 

4. Scans surveys into a computer system; 

5. Analyzes data and interprets for use in a new set of job skills for training; 

6. Writes final report for distribution; 

7. Utilizes data for training workshops; and 

8. Evaluates user impact. 

The use of occupational analysis primarily includes four applications: 

1. Personnel — merging or separating occupations; 

2. Training — what, when, where, to whom, and how to accomplish; 

3. Promotion testing — automated test outlines; and 

4. Research— contribution to the occupational research data bank. 

Six products from the United States Air Force Occupational Analysis System include: 

1. Comprehensive task listings; 

2. Calculation of the percentage of personnel performing various jobs; 

3. Work and task structures for each viable United States Air Force occupation; 

4. Training assessment information; 

5. Testing; and 

6. Task analysis information, including knowledge and skills required. 

According to Tartell (1993), the Air Force, along with Dr. Raymond Crystal, through extensive 
research and development, established the CODAP system in the 1960s and 1970s. In the late 1970s and 
the early 1980s, the United States Navy, the United States Marines, and the United States Coast Guard, 
with Dr. Douglas Goodgame, undertook reprogramming CODAP for their usage on IBM equipment. Since 
that time, those service branches have sponsored many revisions (Woods and Davis, 1993). 

This CODAP system allows the three services to use a common core of processes for validation, 
merging of tasks, adding new requirements, and creating new jobs from the vast array of skills banked in 
the system. The system allows for skills in job families common across the three services to be processed 
in the same manner so as to create a common interpretation about tasks required of similar occupations. 
The Navy uses the CODAP system to aid them in the classification processes, determination of new job 
requirements, ratings, testing, and promotion of personnel. Although this provides a base of commonality, 
some processes differ. 
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AH three use a similar occupational analysis system which includes analysis, design, development, 
implementation, and evaluation. 

Analysis includes validating the training requirements through surveys to incumbent workers. 
These surveys are then processed through CODAP. Through this process, current skills are verified, 
merged with other skills, combined with those identified as new by incumbent workers, and merged into 
the classification to design a new list. 

The mock list is then presented to subject matter experts to review for the content. A survey is 
sent to a stratified sample of Marines in the job classification in question for their review. All of this 
information is optically scanned into the CODAP system. Some of the analysis done by the CODAP system 
at this time includes determining if several jobs are required to do the same tasks or if a job has had 
significantly new tasks added. A survey is developed to be sent to another cut of Marines for their 
verification of the tasks in the classification. 

. The new task lists are then developed into training standards with steps to be performed, 
prerequisites identified, terminal learning objectives established, and the support requirements for the 
training set. Each branch of service has a system of written documents, sometimes referred to as 
"Orders." These written documents are used to develop curriculum for the training. The Orders identify 
what will be learned on the job and what will be presented in formal school training. Training may be 
provided by another command. After the training is implemented, personnel are immediately deployed to 
the specifically trained job in order to guarantee the maximum level of proficiency in the skills acquired. 
The evaluation of the training uses the skill standards to determine how well the personnel are performing 
the tasks on the job. 

The United States Navy performs occupational analysis to update the subject matter for their 
training programs, for testing, for promotional purposes, for ratings of positions, and for creating 
occupational standards. The Navy updates their occupational skill standards every three years. The 
process includes using subject matter experts to set the first draft of skills, followed by a panel of experts 
to review the list. Based on this input, the list is revised and formatted for job incumbents to validate the 
lists. The job incumbents review the lists to ensure that the "real world" tasks are the primary focus. 

The final list is then divided between those skills to be taught on-the-job and those that will require 
schooling. The standards are used to establish advancement exams and study guides for promotion, norm 
referenced testing processes, regulation, instructions, and directions on how to process work. The Navy 
also uses the systemic approach with analysis, design, developnr^nt, implementation, and evaluation. 

The Army is currently examining the possible use of civilian vocational-technical training for 
personnel. If this is to become reality, occupational skill standards will be the communication device. A 
critical area will be the testing and certification of these civilian trained military forces. 
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Education 

Job analysis primarily has been used by vocational-technical education to identify the skills 
students should have mastered upon completion of job training programs. 

The practice of using job analysis techniques by education for this purpose had its early beginnings 
in the 1930s and 1940s as the task analysis methods of industry were utilized in the trades areas. These 
analysis methods were most frequently used in evaluating jobs for which training programs in secondary 
education were linked with apprenticeship programs beginning after completion of high school. This 
practice of job analysis subsided in the 1950s as more emphasis was placed on academic concepts. After 
World War II, many students attended college under the G.I. Bill, and less emphasis was placed on learning 
a skilled trade. It was not until the early 1970s that job analysis techniques began to resurface in new 
formats for use in the schools. 

The primary purposes in this period were to: 

1. Establish measurable vocational-technical education objectives, 

2. Develop curriculum for vocational-technical education, and 

3. Design programs which would prepare students to meet the new requirements of business and 
industry. 

In further developing the occupational analysis process in the 1970s, it was determined additional 
emphasis was needed in the measurement of skill achievement. Prior to this time, educators used 
conceptual models which focused on the processes of learning in an abstract manner. Dr. Robert Mager is 
credited with refocusing education in the early 1970s on performance objectives as opposed to learning 
activities which may or may not lead to learning outcomes. For example, instead of learning math in an 
abstract word problem, students might learn a geometry lesson by determining the appropriate angle at 
which to cut a board being used in the construction of a house. This outcome-based approach focused on 
what was to be attained in measurable terms. When education began to examine outcomes as a basis for 
structuring curriculum, achievement in the real world became more connected to what was learned in high 
school. In other words, the needs of business and industry became connected to education outcomes. 
Evaluation by educators of student performance or mastery of specific skills was finally possilbe. 

Job analysis processes were established by state agencies, large local school districts, and multi- 
state consortiums. This effort was further encouraged by the federal Vocational Education Act of 1964, 
which set aside funds for program improvement in the areas of research, curriculum development, and 
professional development. Also, the national curriculum network received funding to establish a 
dissemination center for each region in the country to assist states in sharing materials and practices. With 
access to these program improvement funds, states had the opportunity to develop processes which would 
fit their individual needs in curriculum development and in-service education. As these efforts began, it 
became apparent that collaboration with business and industry would be necessary to update the 
foundation of the curriculum to reflect the tasks and skills required at the work site. 
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Various practices grew through state efforts, especially in the southeastern United States and a 
few industrialized northern states. These states held a huge investment in area vocational schools which 
were designed to be independent sites dedicated to vocational and technical instruction, primarily for 
upper-level secondary students and adults. 

During this same period of time, several job analysis efforts moved forward. The competency- 
based movement was supported by the National Center for Vocational-Technical Education at Ohio State 
University. The center held workshops and provided technical assistance for designing competency-based 
programs. Competency-based instruction and curriculum was widely accepted in vocational education as a 
means of establishing measurable objectives. 

The national center also began working with the use of DACUM. In part, as a result of the center's 
work today, at least 38 states use this process or some modified form of DACUM as their method of job 
analysts. 

The Vocational -Technical Education Consortium of States job analysis method (discussed below) 
is used by over half of the states At least one major curriculum development consortium, Mid-America 
Vocational Curriculum Consortium (MAVCC), has begun working in a collaborative effort with V-TECS. 

Although some states occasionally have used the interview and Position Analysis Questionnaire 
(PAQ) methods with business and industry projects, few have adopted these methods as their formal job 
analysis process. This may be due in part to the expense often associated with such intensive methods. 



(V-TECS) 

In 1973, several southern states formed a network to develop a common process of task 
development for their states. The network became known as the Vocational-Technical Education 
Consortium of States (V-TECS). 

The consortium has established a systematic development and implementation process for 
competency-based vocational-technical education which includes: 

1. Analysis of jobs and organization of job-related information; 

2. Development of components for assessing student achievement; and 

3. Design, development, and identification of instructional materials that provide a validated link 
between instruction and employment. 

The V-TECS processes develop the following products: 

1. Duty and task lists, including a validated list of tasks performed on the job. 

2. Performance objectives for each task including: 

a. Conditions for performance; 

b. The performance expected; and 

c. The standard as an observable measure of performance 

3. Performance steps detailing sequential steps to perform a task. 

4. Enabling competencies. 

e 68 
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5. Related academic skills providing the language arts, mathematics and science skills utilizing the 
basic essential skills of taxonomy. 

6. Criterion-referenced test item banks: 

a. Cognitive items; and, 

b. Performance or psychomotor domain items 

The V-TECS system has developed worker-validated tasks and performance objectives which 
contain job-based standards of performance for more than 170 occupational domains representing more 
than 700 Dictionary of Occupational Titles. 

Thirty-two states have been members of V-TECS at different periods of time. In addition, all 
branches of the military are associate members, as are the Department of Labor and the Bureau of Prisons. 
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A. INTRODUCT50N AND BACKGROUND 

Testing and assessment are virtually a way of life in modern industrial societies. It begins within a 
few minutes of birth when a doctor strokes her fingernail against the bottom of a newborn's foot to check 
for reflexes and looks for enzymes from a small piece of fingernail. Individual testing continues in regular 
checkups, with assessment of height, weight, teeth and eyes. Kindergarten and elementary teachers 
advance the process by dividing students on teams according to size, strength or speed; Dy testing the 
knowledge of spelling words, and by grading student work. In fact, testing and assessment is integral to 
the American way of education as students are classified and graded according to how well they perform 
on standardized and teacher made instruments. Life choices are, in part, governed by the scores that one 
achieves on standardized tests. In many states even earning a high school diploma is contingent upon 
completion, not simply teacher-made examinations, but also a state level minimum competency tests. All 
of the testing eventually leads to selection of individuals into occupations, often based on how well 
individuals achieved with their grade point average or on selection tests. And, yet the examples only 
represent individual assessment. 

In fact, assessment and testing is fundamental to industrial societies. Goods and services routinely 
are assessed for quality assurance. Programs routinely are evaluated to determine evidence of 
effectiveness and to generate information on which to make decisions. Products are assessed to ensure 
safety and to look for "best buys". 

Assessment and testing is fundamental to any conception of a national standards program. 
Standards programs in other industrialized nations include assessment as a necessary step to certify the 
skills and knowledge of individual trainees. In addition, the information is necessary to determine the 
effectiveness of the training programs as well as the accuracy and usefulness of the standards program 
itself. Hnally, testing and assessment is a necessary ingredient to permit the match of pools of skilled 
workers and manpower needs within the larger labor market. 

Therefore, the purpose of this chapter is to lay out the issues surrounding assessment as part of a 
national standards program. The information is presented in three sections in addition to this introduction: 
legal issues; technical considerations and guidelines for test development; and a brief review of selected 
practices. 
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B. LEGAL ISSUES 

One must consider legal issues associated with the development of national standards and 
certification systems for at least three reasons. First, we live in a litigious society. It has become 
axiomatic that if you have an assessment, testing, and certification program, then someone will sue you. 
Therefore, one must consider the legal issues to prepare for the inevitable. Second, federal and state 
legislation, as well as a large body of case law, acknowledge the appropriate oversight, direction, and 
intervention of law in the development of assessment instruments that are used for making any type of 
employment decision. Third, consideration of the legal issues surrounding assessment helps to emphasize 
the necessity of using rigorous methods for developing and using assessment systems to make selection, 
promotion, termination, certification and licensing decisions within industries. 

Four general areas of legislation, together with their case law seem to apply to the issue of 
assessment. Perhaps the most important law within the assessment arena is the Civil Rights Act of 1964, 
as amended, together with its Uniform Guidelines on Employee Selection Procedures and the accompanying 
Questions and Answers on Employee Selection Procedures. This law, the accompanying regulations, and 
the case law associated with this legislation address the broad issue of the design of tests and 
assessments. A second area of legislation of concern is a combination of the Fifth (for federal purposes) 
and Fourteenth (for State purposes) Amendments. These amendments address the issue of the use of 
information provided through assessments by establishing the right of due process and the expectation that 
governmental agencies will protect the public by policing practice within occupations. A third area of 
legislation directly relevant to the issue of assessment is the Americans with Disabilities Act of 1990, 
which is most concerned with the administration of tests and assessment. The fourth area of legislation 
associated with assessment is the National Labor Relations Act of through which the employees rights to 
be involved in creating the specifics of testing is established. Each of these areas of legislation and case 
law will be explored in the following narrative. 

Civil Rights Laws 

Title VII of the Civil Rights Act of 1964 is the federal statute that prohibits discrimination in 
employment. Specifically, it says that it is unlawful for an employer to refuse to hire or discharge an 
individual, or otherwise discriminate against an individual with respect to compensation and conditions of 
employment, because of an individual's race, color, religion, sex or national origin. Further, and equally 
important, the act states that it is illegal or unlawful for an employer to "limit, segregate or classify 
employees or applicants for employment in any way which would deprive or tend to deprive any individual 
of employment opportunities or otherwise adversely affect his status as an employee because of such 
individuals race, color, religion, sex or national origin." Case law associated with this legislation has 
established two principles through which to view discrimination: disparate treatment and disparate impact. 
Disparate treatment is intentional discrimination where an individual proves that an employer rejected him 
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because of race, religion, national origin or sex. Disparate impact does not address employer intent, rather 
disparate impact occurs where neutrally appearing selection criteria or procedures have the effect of 
discriminating against a protected class. The effect is called adverse impact. It is in light of adverse 
impact that assessments associated with any national standards and certification must be examined. 

The Uniform Guidelines on Employee Selection Procedures and the Questions and Answers were 
developed by the U.S. Equai Employment Commission to assist employers to create selection procedures 
that would be legal within the confines of the Civil Rights Act of 1 964. The uniform Guidelines define 
discrimination and adverse impact, prescribe how adverse impact is determined, establish standards for 
conducting validity studies, and indicate the type of documentation that must b< collected and maintained. 
While the uniform Guidelines and Questions and Answers are not enforceable os law, two previously 
mentioned Supreme Court cases, Griggs v. Duke Power (1971), and Albermarle Paper Company v. Moody 
(1975) established the precedents of providing "great deference" to the guidelines and questions. In 
Griggs v. Duke Power , the Supreme Court struck down employer use of tests because the tests 
disproportionately excluded black applicants from acquiring new jobs within Duke Power. In Albermarle 
Paper Company v. Moody , the Supreme Court adopted language from the uniform Guidelines in 
establishing the standard for employee testing. The Court held that when disproportionate members of a 
protected class fail to perform adequately on a test, employers must show "by professionally acceptable 
methods that the results of the test are predictive of or significantly correlated with important elements of 
work behavior which comprise or are relevant to the job or jobs for which the candidates are being 
evaluated". In other words, if a test has adverse impact, the test must be valid. 

The uniform Guidelines set forth requirements that must be followed in making any type of 
employment decision. For example, the uniform Guidelines state that they "incorporate a single set of 
principles which are designed to assist employers, labor organizations, employment agencies and licensing 
and certification boards to comply with requirements of the federal law prohibiting employment practices 
which discriminate on the grounds of race, color, religion, sex and national origin. They are designed to 
provide a framework for determining the proper use of tests and other selection procedures." Moreover, 
they address any single measure, combination of measures, or procedure that is used as a basis for any 
employment decision. Further, they claim jurisdiction over the full range of assessment techniques, 
including paper and pencil tests; performance tests; work sample tests; training programs; probationary 
periods; physical, educational and work experience requirements; inform or casual interviews; unscarred 
application forms, performance appraisals; and any other technique used for making such decisions. 

Without getting into an elaborate discussion of specific cases, the case law that has evolved 
through the application of Title VII suggests a series of guidelines that must be applied in developing tests 
and assessments: 



► An employer's test must be based on a thorough job analysis that establishes the knowledge, 
skills and abilities necessary to perform successfully the job that is under consideration. 
( Albermarle Paper Co. v. Moody 1975; United States v. the City of Chicago , 1977; Kirkland v. the 
New York Department of Correctional Services , 1974). 




► Evidence must be produced to demonstrate a link between the job task analysis and the test 
content to demonstrate test validity. ( U. S. vs. the City of Chicago , 1984). 

► The content validity of the test must include results from the job analysis that indicate the 
relative importance of the identified work behaviors. Furthermore, there must be a correlation 
between the importance of a job function as determined by the job analysis and the weight it is 
given to this function on the examination. ( Easley vs. Anhevser-Busch, Inc. , 1983; U.S. v. the 
City of Chicago , 1978). 

► Tests must provide for adequate agreement among raters of behaviors of job applicants. 
( Thomas v. the City of Evanston , 1985). 

► Records of the job analysis and of test results must be accurate and kept on file as future 
documentation. In addition, the employment decisions that are rendered must be based on the 
documented evidence produced by the test ( Biqbv v. the City of Chicago , 1984; Brito v. Zia 
Company, 1973; Loiseau v. the Department of Human Resources of the State of Oregon , 1983; 
and Marquez v. the Omaha District Sales Office of the Ford Motor Company , 1971). 

► Ratings that are established should be behavioraliy based on performance dimensions rather then 
personality traits. Sample of traits that the courts have ruled against include items such as: 
attention to detail, interest in the job, temperament, adaptability, general business acumen, 
leadership ability, personal appearance, cooperation and dependability. ( Young v. Edgecomb Steel 
Company, 1974; Robinson v. Union Carbide Corporation , 1976; James v. Stockholm Valves & 
Fitting Company , 1977). 

► Assessment checkout must adhere to a series of criteria for content validity that must be 
followed when constructing any type of performance assessment. These criteria include the 
following: (a) testmakers must have conducted a suitable job analysis; (b) testmakers must have 
used reasonable competence in constructing the test; (c) the content of the test must be directly 
related to the content of the job; (d) content of the test must be representative of the content of 
the job and (e) the test must be used with a scoring system that usefully selects among the 
applicants who can better perform the job. ( Guardians Association of the New York City Police 
Department v. the Civil Service Commission, 1980 and Vanguard Justice Society v. Hughes , 



Although these findings provide useful direction when constructing assessments, at least two 
problems remain. First and unfortunately, even if an industry follows these guidelines to develop selection 
procedures, there is no absolute guarantee that the courts will find the test procedure valid. Kleiman and 
Faley (1978) analyzed court cases as well as the Guidelines and federal law, concluded that "the courts 
have not agreed upon a uniform set of standards. In some instances, tests are judged solely on the basis 
of their face validity; in others, extensive evidence is required. The major factor accounting for these 
differences appears to be the philosophy and/or sophistication of the judges with respect to the area of 
personnel testing." 



1984). 
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Second, it unclear that the principles of validity absolutely pertain in the absence of adverse 
impact. The Guidelines indicate that validity is questioned only if adverse impact is found. Nevertheless, it 
would seem that in the system of national standards, the issue of valjdity should take precedence 
regardless of the existence of adverse impact. 

Labor Relations Laws 

A second legislative area that has implications for the development of assessments is the National 
Labor Relations Act. The National Labor Relations Act applies in unionized workplaces and specifically 
requires that the employer and the union bargain over wages, hours and terms and conditions of 
employment. The Act relates to the development and use of assessment techniques in a standards and 
certification program because case law has established the right of employees to have a voice in the 
creation and use of such assessments. For example, in the Rapid Roller Company v. The National Labor 
Relations Board , the courts held that employees and unions must bargain over procedures used to establish 
the promotion system. Furthermore, in Phelps, Dodge, Cooper Products Corporation v. The Nation al Labor 
Relations Board , the courts held that the employer and the union must bargain over procedures used to 
provide merit raises. In both instances, the law indicates that in the organized sector, labor unions must be 
included with equal voice in the creation of standards and formal systems including assessment systems 
through which employment decisions are made. 

Disabilities Laws 

A third area of legislation with direct implications for assessment within a program of national 
standards is the American Disabilities Act of 1990 (ADA). Among the provisions of the ADA are several 
that pertain to agencies that give or use the information generated by assessments and examinations. Title 
II of the ADA prohibits discrimination in offering benefits, programs or services by state and local 
governments. Title III of the ADA prohibits discrimination in the offering of public accommodations and 
services, and specifically states "any person that offers examinations or courses related to applications, 
licensing, certification, or credentialling for secondary, post secondary education, professional or trade 
purposes shall offer such examinations or courses in a place and manner accessible to persons with 
disabilities, or offer alternate accessible arrangements for such individuals." Thus any standards process 
that involves assessment and certification is prohibited from discriminating on the basis of disability. 

ADA guidelines have been developed to cover a number of the responsibilities of certification 
boards. For example, applications for testing must include language that allows someone with a disability 
to indicate that they have a specific need for some type of an accommodation during the testing program. 
Further, while it may be appropriate for the testing agency to ask for documentation of a particular 
disability to ensure the validity of the request for accommodation, the agency or entity administering the 
examination is required to bear the cost for any modification or aids provided within the examination. 

(4 
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The documentation of the disability should be provided by a competent professional familiar with 
the applicants disability, and must be treated as confidential and privileged information. If an agency 
cannot provide the requested accommodation at its primary location, the agency may work to obtain 
alternative sights, or may provide an appropriate alternative accommodation. 

The test itself must not assess the disability, but rather skills and knowledge appropriate to the 
job. That is, very often, a written paper and pencil test might have been designed as a timed test even 
though time on such as task is not related to the job. Therefore, under ADA accommodations, the test will 
not include a time component. However, a work sample test whose time component accurately reflects 
the requirements of that job in the field, may include a time component. 

Among the type of accommodations that a test giver should consider are at least the following: 
architectural accessible testing sites; distraction free space; test schedule variation; extended time for the 
test taker; a scribe to assist in recording answers; sign language interpreters; readers; modification of the 
test presentation such as braille editions or large print editions; and adaptive equipment. However, the 
most important characteristic is that the assessment itself must accurately reflect job requirements rather 
than some criteria more accurately associated with the individual's disability then with the work. 

A last area of legislation related to assessment within national standards and certification systems 
relates to the legal basis for licensing. The whole issue of due process and licensing examinations is 
important to the consideration of national standards and certification as an area of law for three reasons: 
First, if government takes a lead in establishing national standards and credentials that limit people's entry 
into or advancement within certain kinds of occupations, then the entire process may be subject not only 
to Title VI! of the Civil Rights Act claims, but also to "equal protection and due process" claims under the 
5th and 14th Amendments. Second, it is important because whereas licensing exams have not been held 
to the same standard of validity that certification exams have been held, particularly with regard to Title VII 
of the Civil Rights Act, if a public entity uses the results of a licensing test to select (as opposed to 
categories) qualified persons for employment, then, in fact, Title VII applies to the situation ( U.S. v. South 
Carolina , 1978). Third, the issue is important to note because the failure of the courts to apply the 
rigorous validity standards used for employment examinations to state licensing examinations is simply 
hypocritical. Licensing examinations are a type of prerequisite to professional employment and seem like 
they should be treated like any other. In a national system of industry standards and certification, the 
similarity between certification assessment and licensing exams is bound to demonstrate the importance of 
treating both systems identically. 

The law holds that a government may regulate entry into and the practice of some occupations in 
order to protect the public. Constitutional due process established by the 5th and 14th Amendment, 
however, requires that the power of the state be exercised only where it is necessary for the health, safety 
and protection of the general population. The voluntary adoption of skill standards and certification does 
not constitute a licensing scheme, nor does licensing preclude the adoption of voluntary standards within 
an occupation, if employers and unions make decisions about the standards and the criteria for 
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certification in making hiring decisions, the issue of due process in licensing does not apply. However, to 
the degree that any government agency takes a leading role in setting standards that become prerequisites 
to employment or advancement in some occupations, then the relationship between the licensing, 
certification and requirements of due process becomes much more important. 

To date, the courts have held that "a state can require a high standard of qualification, but any 
qualification must have a rational connection with the applicant's fitness or capacity to practice [a licensed 
occupation]" ( Schware v. The Board of Bar Examiners , 1957). The question is, does the standard 
rationally related to a legitimate state purpose and does it consistently apply? 

Most courts have upheld licensing standards if the court can think of any rational purpose to 
support the requirement. 

Due process relates to the issue of certification and licensing to ensure that: (a) all applicants are 
treated identically and (b) that the course and test content reflects standards on which graduation, 
certification, licensing and other types of formal recognition are based. Just as courts have ruled that 
students have a property interest in receipt of their high school diplomas, that psychologists have a 
property interest in receipt of their license, so too, the courts unquestionably will rule that workers, 
tradesmen, and others whose jobs are included in some sort of standards and certification programs also 
have a property interest in the certification that results from that program. This means that in a standards 
and assessment program, trainees are required to receive adequate notice of expectations and an 
opportunity to be heard through a fair test. Fair testing requires a direct relationship between the 
curriculum materials, job expectations, and the content of the examinations. This content and criterion 
referenced validity to testing clearly is one of the situations that must be included in consideration of the 
legal implications of assessment. 

C. TECHNICAL CONSIDERATIONS AND GUIDELINES FOR TEST DEVELOPMENT 

The technical standards for test development are contained in Standards for Educational and 
Psychological Testing (1985) (American Psychological Association, the American Educational and Research 
Association and the National Council on Measurement in Education). Any type of assessment that is used 
for making employment selection and promotion decisions is covered under these standards. The following 
12 guidelines emerge from the standards: 

1 . Test content and performance criteria must be based on a thorough job analysis that specifies 
the tasks to be performed and/or the knowledge, skills, abilities and other personal characteristics 
necessary to do the work. In addition, the ratings of relative frequency and importance of the 
tasks, knowledge, skills and abilities must be specified within that job analysis. Job analysis is the 
only means through which to tie the content of the job with the content of training and 
assessment. 

2. One must prepare a specification document in which is set forth all the rules for developing the 
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assessment. These rules should include, at a minimum, an outline of the content of the 
examination; the relative weight of different parts of that content for the examination process; the 
rules for writing test items including not only traditional multiple choice, true or false, oral and 
essay questions, but also work samples on the job assessment situations; and other types of 
performance evaluation. Usually the Specification Manual also should include guidelines for 
validating the items and general directions for administering, scoring and reporting the assessment. 

3. Competent persons i-eed to be involved in developing the test items and situations. Persons 
with direct knowledge of the occupation must be involved in the review and necessary revision of 
initial assessment items and situations. The review must discuss item and situation format, 
directions, placement within the overall document, and the most correct answer for the item. 
Questions also must be reviewed to insure that they do not inadvertently advantage or 
disadvantage any subgroup population. Lastly, the reviewers should confirm assignment and 
correspondence of a question or situation to the specifications within the Specification Manual. 
That is, they need to insure that the question or assessment situation accurately reflects the type 
of job task that is knowledge or skill that is being assessed. Notes on the entire review process 
should be kept as part of the overall documentation of assessment development. 

4. One must assemble the examination form, from the items and situations created in the 
assessment pool and based on the directions provided in the Specification Manual. Attention must 
be given to the placement of items and situations within the assessment. Further, once completed 
the entire assessment process must be reviewed by content experts to determine that the 
situations and questions work, that the pattern of responses is correct, and that the overall 
sequence of information makes sense. Further, the test constructors must devise a set of 
directions to guide the trainee through the situation or assessment and answer questions that they 
might have. The set of directions must be standardized across all test takers. 

5. One must develop criteria measures to differentiate between those that succeed in the 
assessment and those that do not succeed in the assessment. While the guidelines do not 
prescribe the development of criterion measures, they argue that the measures should not be 
arbitrary (based on intuition) be based on an unrelated standard like in school (where the minimum 
passing grade is 70%, and is established by fiat for the entire school system), nor be based on 
quotas (where a set percentage or number of trainees pass and an additional number fail similar to 
grading on the curve in order to ensure that certain numbers of students remain in all categories of 
grades). 

Instead, one should select criterion measures based on other^ypes of qualities. Among the 
measures that one might consider are measures of accuracy within tolerance limits; the number of 
allowable variations or permissible errors; the quantity or rate of production within an assessment 
situation; the adherence of products and answers to some criteria included in reference documents 
that insure the criterion references of the examination; the time limit or speed with which certain 



77 



69 

types of procedures are completed successfully at work and in the assessment situation; the 
amount of supervision or assistance that might be provided in the assessment situation or in the 
performance situation; and the quality or degree of excellence in the effort; and the goodness of fit 
between the work product within the assessment situation and the work product actually involved 
on the job. 

Each of these criterion measures, while not prescribed in the guidelines, more closely approximates 
what actually occurs at the job than does traditional grading. In fact, the criterion measure should 
reflect the necessary requirements to be successful at work rather then some standards merely 
created for the assessment situation. 

Often experts are used as a panel of judges to establish the criteria for success within the 
assessment situation. The guidelines specify that when professional judgement is used, the 
qualifications of the judges should be documented. Unfortunately, the guidelines do not suggest 
how to choose judges nor how many is appropriate. Yaeger (1991) has investigated these issues 
and offers the following suggestions. Specifically, Yaeger suggests that judges should possess an 
organized knowledge base on the content of the topics under assessment. Further, they should be 
chosen on their ability to conceptualize successful and unsuccessful candidates within the 
assessment situation, and to explain what those differences might be. Further, they should be 
able to accurately judge the difficulty of the problems at hand, and be able to sort and 
conceptualize those problems into categories. 

Yaeger also addresses the issue of how many judges are sufficient to determine the criterion 
measures. He suggests using a formula that relates the estimated standard error of the mean, the 
standard deviation and the number of items. 

6. One must establish procedures for insuring fairness in the treatment of those being assessed. 
These procedures should include, at a minimum, information about expectations on the 
assessment; how the scores on the assessment will be reported and used; the types of questions 
or situations included in the assessment; practice questions; application information; and other 
similar types of information. This information must be prepared to share with assessment takers 
before the testing situation. Many times, this information is available in a testing bulletin or in 
"practice" test books that can be purchased. This standard also requires an application form that 
adheres to the rules of the ADA. 

7. One must train raters and establish inter-rater reliability. When a work-sample performance 
examination is included, a performance can be viewed only once. Because judges may differ in the 
way they view an observation, at least two judges should view any candidate's performance. The 
judges should make independent observations, and reach a decision about the appropriateness of 
the performance. In addition, the ratings of the two judges should be compared to insure inter- 
rater reliability and training should be provided so that any pair of judges would view the 
performance in the same way. 
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8. A Test Administration manual should be provided that insures a standard operating procedure 
for giving the assessment. This manual should include information, at a minimum, on entry and 
exit during admission, on how to handle irregularities, introduction to the testing situation, scoring 
information, and the discussion of feedbt ,'■ Time limits as well as adaptations to meet 



9. Directions must be provided on analyzi. d reporting the information. This means providing 
directions on how to score the test or assessment, how to compare the scores against the 
standard or other scores, and how to report that information, both to the candidate and to the 
credentialling body. Other provisions of these directions should be instructions on storing the 
information, on responding, to inquires about text scores, and documentation about the test that is 
available to this public. 

10. One must provide explanations of formal procedures for test analysis and review. Three 
issues must be addressed in this standard guideline. First, the test developers must provide 
evidence and directions for continuing analysis of test items and assessment situations. This 
analysis includes in determining the difficulty level and the discrimination indices for each of the 
items. In addition, it should address the reading level of the items as well as their adherence to the 
guidelines for item writing. This requirement holds whether it be a paper and pencil test or any 
other type of assessment. Best practice should be followed as one develops this type of 
information and it must be maintained by the agency on whose behalf the test is provided. 

The second part of providing procedures for test analysis and review is to analyze the scores and 
what they mean, by groups of individuals who actually take the test. That is, one must assess, by 
types of characteristics such as educational level, job, performance ratings, and membership in 
groups such as minority groups to insure that no adverse impact is being produced by the test. 
One must be particularly sensitive to demonstrate that the test scores, particularly the high and the 
low scores, in fact, have a direct, continuing relationship to performance on the job. 
The third portion of providing for procedures for test analysis and review is to generate guidelines, 
suggestions and time tables that the test developers and sponsoring organization should follow in 
updating and reviewing test items in situations to insure that they remain current. Such guidelines 
need to address new technology within the industry as well as advances in the testing procedures. 

11. One must provide for examination security. The sponsoring organization as well as the test 
developers must make security arrangements to keep private, the assessment instrument, the 
scoring, and the individual scores that are generated through taking the test. Confidentiality laws 
hold in this situation. Clearly any breech of security results in disallowing use of the test for 
selection purposes. Test scores of individuals need to be maintained for a given period of time as 
determined by the sponsoring organization. Usually that period of time is at least three years and 
more likely 5 to 10 years. 



requirements of ADA should be included 
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12. One must develop and maintain specific information on each type of validity necessary to 
support the test. The guidelines and standards specifically require developing three types of 
, y validity information, depending on the type of testing. The three types of validity, as explained 
elsewhere in this document, are content validity, criterion-related validity and construct validity. 
Depending on the type of assessment, information supporting each of these types of validity for 
that assessment instrument must be developed and maintained in clearly understandable terms by 
the test sponsoring organization. In addition, other types of validity such as concurrent, predictive 
and face validity may be maintained but are not sufficient to support either technical or legal 
challenges to the test. 

D. INDUSTRY PRACTICES IN ASSESSMENT 

Businesses and industries use many types of procedures as assessments in order to make 
employment decisions about candidate selection and promotions. Riley and Warech, in considering the 
validity and fairness of different assessment measures, suggest that there are at least 18 different types of 
assessments used by American employers. These types of assessments are illustrated in Table 1 . As 
comprehensive as this list is, it ignores two methods frequently used by employers: on-the-job assessment 
of actual practice which is frequently used in apprenticeship programs; and portfolio assessment or the 
intentional inspection of work products against some criteria to determine proficiency. Each of these two 
types of assessment is addressed later in the chapter. 

Table #1 
Types of Assessments or Tests 
Used by American Employers* 



1. 


Paper and Pencil Tests 


2. 


Work Samples 


3. 


Assessment Centers 


4. 


Trainability Tests 


5. 


Interviews 


6. 


Peer Evaluations 


7. 


Self-Assessments 


8. 


Bio Data 


9. 


Grades and Educational Achievement 


10. 


Reference Checks 


11. 


Seniority Systems 


12. 


Personality Tests 


13. 


Projective Techniques 


14. 


Individual Assessment by Experts 


15. 


Scored Training and Experience Evaluation 


16. 


Handwriting Analysis 


17. 


Tests of Physical Ability 


18. 


Integrity Tests 



•Source: Richard R. Reilly and Michael A. Warech. (1988). The Validity and Fairness of Alternative 
Predictors of Occupational Performance . Berkeley: Center for Behavioral Studies at the University of 
California, 1988. 

ERLC oO 



72 



Historically, employers have relied on the paper and pencil type tests as their primary means of 
assessment. Dependence on this technique is reasonable given the development in area of the field of 
psychometrics in which paper and pencil tests predominate. Further, because these are the types of tests 
that trainees are used to taking in school and the types of tests that people in industry took when they 
were in school, it is a logical step to continue their use in making employment selection. The Uniform 
Guidelines and Questions and Answers related to employment selection address specifically the design and 
development of paper and pencil tests as opposed to other types of measures such as work samples. 

Use of paper and pencil tests in employment and training settings has spawned an entire support 
industry that has been used to assist in developing and administering those tests to make employment 
decisions. That support industry is available to provide assistance to the development of assessment for 
certification. It includes for example, a number of companies and organizations whose specific job is to 
develop tests. For example, the Educational Testing Service, the National Occupational Competency 
Testing Institute, the American Institute for Research, the Council on Occupational Research and 
Development, the American College of Testing, all are organizations whose purpose is to design and 
administer tests useful for selection purposes for employers in training institutions. 

When one examines the paper and pencil tests that these organizations have created, one 
discovers that they have done a good job of conforming to the standards as established by case law and 
the Uniform Guidelines . Usually, the items are based on a formal job and task analysis that often involves 
job incumbents identifying specific tasks. Always, the tests are developed around Specifications Manuals 
that link the items to the tasks, and the tasks to the domains of activity within the occupation. The 
Manuals also assign relative weighting, by number of items, that are associated with the various tasks. An 
example of a page from such a Specification Manual recently developed for the National Glass Association 
is included as Figure 2. 

Test writers usually are careful to devise and follow specific guidelines for writing test items and 
performance situations. These rules are developed from research on "best practice" in item writing and 
typically are articulated in the Specifications Manual. A sample of the type of item writing directions that 
might be provided in a Specification Manual is included as Table 4. Notice in the Table that the specific 
language of items, the length of items, and their relationship to each other are addressed within the rules. 

The professional test organizations also are very careful in establishing statistical measures of 
reliability and making that information available to the industry association or employer who may be paying 
for the assessment. Often reliability is calculated through one or more accepted practices such as test- 
retest, or split half coefficient. Unfortunately, too often reliability, and not validity, seems to be the focus 
of primary professional test developers. 

When one examines sample certification test items prepared by professional organizations, one 
discovers some of the strengths and limitations of paper and pencil tests. For example, one can use 
Benjamin Bloom's Taxonomy of Educational Objectives to divide work tasks into different levels of 
functional activity, and then analyze test items in terms of the type of content and intellectual activity that 
they require in order to answer them correctly. Additionally, although Bloom specifies six levels of 
cognitive function from recall through evaluation and synthesis, many of the paper and pencil tests that are 
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Figure #2 
Example of Final Test Specifications 
Senior Automobile Glass Technician* 
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RUBRIC 


DOMAIN 


SUBDOMAIN 


TASK 


% OF 
EXAM 


# OF ITEMS 
(70 TOTAL) 


01 


GLASS INSTALLATION 






79 


55 


0101 




Standard Replacement 




48 


34 


010101 






Inspect Vehicle for 
Previous Damage 


6 


4 


m 01 09 






Project Vehicle 


6 


4 


010103 






Verify Parts 


6 


5 


010104 






Select Tools and 
Safety Equipment 


6 


4 


01 01 or 






Remove Old Parts 


6 


4 


01 01 06 






Clean and Prepare for 
Installation 


6 


4 


010107 






Install New Part 


6 


5 


010108 






Clean and Detail 
v e i ii uie 


6 


4 


A 1 AO 




raiiern ividMny 




4 


3 


A 1 AO A 1 








2 


1 -2 


01 OzOz 








2 


1-2 


0103 




Glass Cutting 




6 


4 


010301 






Glass Selection 


2 


1-2 


01 0302 






Glass Cutting 


2 


1-2 


010303 






Glass Finishing 


2 


1-2 


0104 




Trouble Shooting 




5 


3-4 


010401 






Diagnose Glass Breaks 


1 


0-1 


010402 






Solve Problems Caused 
by Rust 


1 


0-1 


010403 






Locate & Correct Air 
and Water Leaks 


1 


0-1 


010404 






Diagnose Moveable 
Glass Operation 
Problems 


1 


0-1 


010405 






Correct Appearance 
Defects 


i 
i 


0-1 


0105 




Operating Mobile Units 




4 


2-3 


010501 






Install & Repair Glass & 
Related Materials In 
Uncontrolled Weather 
Conditions 


2 


1-2 


010502 






Stock Tools & Parts in 
the Mobile Unit 


2 


1-2 



•Source: National Glass Association, McLean, Virginia, 1993. 
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Table #4 
item Writing Guidelines 

1 . Word questions clearly and concisely. 

2. Avoid negative and double negative wording as well as other types of absolutes. If they 
must be used bold or capitalize the type. 

3. Ensure that questions are capable of standing independent of other questions in the 
examination. 

4. Write stem so that it asks the central question, allowing a candidate to form a tentative 
answer. 

5. Eliminate clues or tricks from the question. 

6. Write the stem longer than the options. 

7. Use key words in the stem consistently. 

8. Avoid unfamiliar, figurative, literary or textbook language. 

9. Eliminate words (some, often) that may have different meanings to different persons. 

10. Arrange options in a systematic order. 

1 1 . Use the option all of the above only sparingly (if used, it should NOT be correct all of the 
time). Avoid it where possible. If none of the above is used, make it the correct option at 
least some of the time. 

1 2. Write stem and options in the same style, with the same type of language. Ensure 

consistency with the structure of the stem (singular versus plural, make versus female). 

13. Write options parallel in length, form and content. 

14. Make all options plausible; make some very nearly the best answer. 

15. Make sure there is only ONE correct option. 

16. Write question grammatically correct and at the reading level of the trainee. 

17. Include visuals where possible. 

18. Eliminate overlapping options. 

19. Write stem as a complete sentence. 
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used for employment seem to concentrate on the more elementary types of intellectual ability. That is, 
there tends to be an emphasis within paper and pencil tests to test for recall or simple recognition of 
information, rather then to look for the application of that information or evaluation of skills. The emphasis 
on recall is understandable given the difficulty of constructing valid paper and pencil tests to deal with 
issues such as evaluation, synthesis, and application. Nevertheless, it is possible to build items that match 
job tasks and intellectual function and some of the certification exams do so. 

To point out the differences, let's examine test questions from two of the assessments that are 
currently used for certification purposes: In one instance, questions are from tests prepared by the 
National Occupational Competency Testing Institute to assess and certify the skills learned by Job Corps 
program participants; and in the second instance questions are from the Automobile Mechanics 
Certification Examination tests prepared for the National Institute for Automotive Service Excellence by 
American College Testing. Examination of a draft of the test for Job Corps participants reveals one of the 
limitations of paper and pencil tests. While the items conform to some of the "best practices" for item 
writing for paper and pencil tests, the items test little of the range of functions that a worker must perform 
on the job and they do not assess those functions in a proportion relative to the amount of time that the 
worker spends doing the different functions (or using the knowledge) on the job. By taking a sample of 
test items, one determines that over 80% of the items on the examination measure what, in Bloom's 
Taxonomy, would be considered recall and recognition skills. That is, the test items simply ask the trainee 
to identify certain types of terms and tools or to recall specific information about materials associated with 
the trade. Relatively few of the questions ask trainees to manipulate information, to apply information to a 
trade situation, to exercise judgement, or to solve problems. In fact, the questions that required more than 
recall tend to be clustered in the mathematics part of the examination. A sample of two of the typical 
questions from the examination are included as questions 1 and 2 in Figure 3, Sample Assessment 
Questions. 

By contrast, an examination of the test to certify automobile mechanics illustrates a much broader 
range of intellectual function when one analyzes the test questions against the different functional levels 
expressed in Bloom's Taxonomy. Not only are recall and recognition questions included, but also questions 
that relate to application of information and diagnosis of information are included. In fact, a sample of 
questions from that practice exam illustrate many rules of "best practice" for item writing for paper and 
pencil multiple choice tests, and suggest that only about half of the items deal with recall and recognition, 
A much larger percentage of items ask the trainee to apply knowledge or skills that they have learned in 
training to an actual job situation. Other questions ask trainees to apply knowledge either to produce the 
answer or to diagnose a problem as described in the test question. Two sample questions requiring higher 
order skills using Bloom's Taxonomy are illustrated as Questions 3 and 4 in Figure 3, Sample Assessment 
Questions. 
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Figure #3 
Sample Assessment Questions 

Sample Recall and Recognition Questions 

1. Identify the temporary structure of platform used to support workmen and materials during 
construction. 

A. masonry table saw 

B. scaffold 

C. mortar board 

D. all of the above 

2. The vertical holes in a block are called. 

A. partitions 

B. cells 

C. webs 

D. lugs 

Sample Questions Requiring Higher Levels of Functional Skills 

3. You are doing a ring job on an engine. The cylinders have .015" taper. Which of the following 
would be the correct procedure? 

A. Bore the engine. 

B. Install expanders behind the new rings. 

C. Knurl the pistons. 

D. None of the above--this is an acceptable taper specification. 
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According to the above chart, if the temperature of the outside air reaching a normal working air 
conditioner is 71oF, the pressure in the discharge side of the system is: 

A. 129psi 

B. 15.8psi 

C. 1 2 psi 

D. None of the above 
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In general, paper and pencil tests for certification have distinct advantages and disadvantages. 
Advantages include allowing sponsors/test developers to provide excellent statistics on reliability, great 
ease of administration, and comparable practice for scoring. However, often they do not accurately reflect 
content validity in an occupational setting because they emphasize school learned information, and deal 
with verbal type of information, and emphasize recall. It is very difficult, although not impossible, to 
assess skills. And they are expensive to do develop although relatively inexpensive to use. 

Historically, some segments of American industry have used other types of assessment techniques 
to certify individuals. For example, in apprenticeship programs, as registered apprentices learn by working 
and earning, they demonstrate their skills and eventually achieve journey worker status. Typically, the 
assessment that occurs in an apprenticeship program involves several activities. First, assessment is linked 
to standards as expressed in the registration document for the industry and as expressed in the 
apprenticeship agreement that serves as a contract for training, pay and assessment between the 
apprentice and the sponsoring agency. A sample copy of standards for the carpentry trade is included as 
Table 5. One will notice in the Table that work experience standards are expressed in terms of broad 
occupational areas of concern such as form building, and expressed in terms of specific tasks and activities 
that an individual must demonstrate such as building and placing straight concrete forms; and irregular 
concrete forms, and concrete forms for stairways, floors, walls and columns. In addition, the standards 
prescribe an approximate number of hours that an apprentice must work successfully on the trade practice 
in order to master the skills. 

Assessment occurs on three levels. First, work hours are recorded and reported to the Joint 
Apprenticeship and Training Committee for each apprentice. Second, the apprentice's ability to do that 
work is recorded because the apprentice must continue to work successfully on the trade practice to keep 
his/her job. By maintaining the opportunity to earn a living, the employer provides the screening that 
insures the standard is being met. Third, in all apprenticeship programs, the apprentice is required to 
attend related instructional courses. Related instruction typically is offered in the classroom and deals with 
the supplemental and theoretical information and knowledge of the craft. Typically, in related instruction 
programs, paper and pencil as well as demonstration tests are required in order for an apprentice to 
successfully complete that course. 

Thus, three types of information usually are reported to the Joint Apprenticeship and Training 
Committee: (a) information on successful completion of topics from related instruction; (b) numbers of 
work hours by type of job process; and (c) information about skill levels mastered either during training or 
exhibited on the job. A typical apprentice on the job evaluation report is included as Table 5. This 
particular report is used by the International Union of Bric'k Layers & Allied Craftsmen to track the skills 
that apprentices learn at work. 

The Joint Apprenticeship Committee reviews the progress of an apprentice every six months and, 
if the progress is satisfactory, advances the apprentice to a new pay grade. 

In most apprenticeship programs, the specific skills that are mastered through the work processes 
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are recorded on an apprenticeship training record that indicates the skills that the apprentices mastered and 
the degree of level of mastery provided. This type of profile is included as Figure 5. This profile for the 
printing industry illustrates two additional points about assessment in apprenticeship. First, note that an 
individual's skills typically are rated on some numeric scale that ranges from "no experience" to "skilled". 
Most often "skilled" means being able to work independently with no supervision. Second, the 
proficiencies or competencies are expressed within broader tasks as expressed within the apprenticeship 
standards document. The Figure depicts a small sample of skills related to several tasks that someone in 
the printing industry must demonstrate: paste up and layout, composition, and reproduction photography. 
Each of those activities then is broken into specific competencies for which an assessment is recorded. 

Although this type of assessment may provide better content validity for the work that one 
actually must perform, it does not necessarily use "best practice" in terms of the rating scale itself. The 
descriptors of "skilled" through "no exposure" with the numeric ratings do not accurately describe the 
levels of proficiency that an apprentice or journey worker may have demonstrated. And the 
scale/descriptors introduce potential for serious inter-rater reliability problems. 

In some trades, several new types of assessment techniques are being used. For example, the 
International Masonry institute, the training arm of the International Union of Brick Layers & Allied 
Craftsmen and their signatory contractors, has developed and is using a portfolio type of assessment to 
certify the skills of their journeyworkers who achieve specialty certification. For example, the 
journeyworker who becomes a certified instructor must, as part of the training and assessment effort, 
demonstrate his or her skills through a series of activities that include presenting: a number of trade 
drawings produced by apprentices. 

Under the tutelage of the instructor; copies of student records, equipment and inventory records, 
and samples of other administrative procedures required within the system; an extensive video tape 
demonstrating a number of information presentation and classroom management skills in the classroom and 
hands-on training areas; a series of original lessons plans and instructional materials that demonstrate best 
practice for creating such products; and written and oral examinations. 

The Laborer's AGC has moved to create a different type of assessment to certify the trade skills 
and knowledge of its members. They are developing a series of work sample performance tests through 
which they will certify the skills of the members within critical occupational task areas in a variety of 
occupations in the jurisdiction of the Laborer's International Union of North America. The work sample 
tests, which the Laborer's are calling "performance tests," require a laborer to demonstrate the skill in a 
simulated off-job-site setting that approximates the work situation. For example, an asbestos abatement 
worker, in order to earn his or her certification, must successfully complete five performance tests in 
addition to a written knowledge test. One of the performance tests is a test on inspecting and donning 
respirators. The test was devised following the requirements expressed in the Uniform Guidelines and in 
the Standards written by the APA. The test deals with perhaps the most critical job skill of the asbestos 
abatement worker. 
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Table #5 

Carpenter Apprenticeship Standards 

1. CARPENTER 

860.381-022 

Work-Experience Schedule 

The following schedule is an example of the type of work experience and training considered necessary to develop a 
skilled and productive worker in the carpentry trade. Within the limits of basic trade requirements, the schedule is adaptable 
to local conditions. 

Approximate Hours 



a. Care and use of tools and woodworking machinery 325 - 500 

b. Form building 780 - 1,200 

Build and place straight concrete forms, irregular concrete forms, concrete forms for 

stairways and floors, walls, and columns. 

c. Rough framing 780- 1,200 

Floor, wall, roof, stair, scaffolding, etc., on both house and heavy construction. Roof 

covering. 

d. Layout 325 - 500 

Batterboards, partitions, doors and windows, box-out in concrete walls. 

e. Outside Finishing 520 - 800 

Application of door and window trim. Fit and sand doors and windows. Application of all 

exterior finishes and related trim. 

f. Inside Finishing 975 - 1,500 

Application of door and window trim. Fit and sand doors and windows. Application of 

baseboards and moldings. Construction and setting cases, wardrobes, stairwork. Flooring. 
Application of hardware and fittings to exterior and interior of building, doors, and windows. 

g. Welding 325 - 500 

h. Plastics and resilients 195 - 800 

i. Acoustics and drywall 650- 1,000 



1 . Ceilings: 

Layout, cutting, assembly, and installation of all materials and component parts. 

(a) Hangers, channels, furring and backing boards 

(b) Bars; main tees, cross tees, splines. 

(c) Stiffeners and braces. 

(d) Ceiling angles or moldings. 

(e) Finish ceiling materials. 

(f) Items of local practices. 

Walls and Partitions: 

Layout, cutting, assembly, erection, and/or application of ail materials and component parts. 

(a) Floor and ceiling runners. 

(b) Studs, stiffeners, bracing, fireblocking. 

SS 
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Table #5 



Carpenter Apprenticeship Standards (Cont.) 



(c) Resilient and furring channels. 

(d) Layout, framing enclosing, and trimming of doorframes, window frames, vents, 
light wells, and other openings. 

(e) Wall angles and moldings. 

(f) Studless and laminated installations. 

(g) Thermal and sound insulation. 

(h) Installation of backing and finish materials. 

(i) Fireproofing of columns, beams, and chases, 
(j) Items of local practices. 

j. Miscellaneous 260 - 400 

Safety, scaffolding, walkways, shoring, sheds, protection, etc. 
k. Asbestos abatement and other hazardous material handling and disposal 65-100 



Related Instruction 

The National Joint Carpentry Apprenticeship and Training Committee encourages the implementation of performance 
based training. The following subject matter should be considered: 

a. Accident prevention, first aid, safety hazards, State and Federal safety codes and regulations. Safety courses per 
the Occupational Safety and Health Administration, U.S. Department of Labor. 

b. Ethics and history of the trade. 

c. Tools and materials of the trade. 

d. Review and application of basic mathematics. 

e. Elementary blueprint reading and freehand sketching. 

f. Foundations: types and layout. 

g. Rough framing: types, plumbing, leveling, and alignment. 

h. Exterior finishes: kinds and use. 

i. Interior finishes: kinds and use. 

j. Roof framing all types and combination. 

k. Stairbuilding and finishing. 

I. Cabinetmaking: on job-site and installation. 

m. Reinforced concrete form construction: materials, types, devices, and their use. 

n. Heavy timber construction, framing. 

o. Welding: gas and arc, vertical, horizontal, and overhead. 

p. Acoustics and drywall: types and their use, layout, cutting, welding, and installing, 

q. Plastics and residents: types and their use, layout, cutting, welding, and installing. 



Total 



5,200 - 8,000 
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Figure #4 
Apprenticeship Reporting Record 
APPRENTICE ON-THE-JOB EVALUATION REPORT 

Apprentice Name 

Period of Apprenticeship: 1 2 3 4 5 6 

The Joint Apprentice Training Committee requests that you evaluate the apprentice when working on your job. The information you provide 
will assist the Committee in monitoring his/her progress in the apprenticeship program. 



CRAFT SKILLS 
PROFICIENCY CODE KEY— L-M-S 

Limited Skill Requires instruction or close supervision to do work. 

Moderate Skill Apprentice generally can perform good work with limited supervision. May not meet all demands of speed and 
accuracy. 

Skilled. Can work independently in accomplishing work assignment. Meets the full demands of speed and accuracy. 
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APPRENTICE PROFILE 



Personality Traits 


1 


2 


3 


4 


Rating 


CO-OPERATlON-Willingness to work 
with others in a helpful way; to do 
teamwork, to carry out assignments 
of instr. 


Contrary, 

Bothers 

Others 


Uncertain 


Helpful 


Highly 
Cooper- 
ative 




APPLICATION AND INDUSTRY- 
Persistency and constancy of effort, 
manner of meeting and overcoming 
difficulties, advantageous use of time. 


Wastes 
Time 


Easily Dis- 
coruaged 


Works 
Inde- 
pendently 


Very 
Energe- 
tic 




NEATNESS AND ORDERLINESS-Care 
apprentice takes of material and 
surroundings, also personal grooming. 


Slovenly, 
Untidy 


Careless 


Tidy 


Very 
Neat and 
Orderly 




RELIABILITY-Punctuality and regularity 
of attendance; manner of handling 
responsible matters; honesty; degree 
to which apprentice finishes what 
he/she begins. 


Not 

Dependable 


Changeable 


Fairly 
Depend- 
able 


Highly 
Depen- 
dable 




INITIATIVE-Extent to which apprentice 
goes ahead without direction and 
works without supervision. 


Requires 
Constant 
Urging 


Requires 
Supervision 


Does What 
He is Told 


Promo- 
ter of 
New 
Ideas 




APTITUDE-Ease with which 
apprentice acquires skill, learns new 
methods and interprets directions. 


Makes 

Many 

Mistakes 


Makes 

Some 

Mistakes 


Capable 


Excep- 
tional 




WORKMANSHIP-Skill of apprentice; 
degree of pride in quality of work 
produced. 


Sloppy 


Many 
Mistakes 


Good 


Excep- 
tionally 
Neat in 
Applica- 
tion 





SIGNED DATE 

Supervisor 
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Level L: 
Level M: 

Level S: 
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The test requires the abatement worker to demonstrate and talk his or her way through a number of steps 
as they perform the test. That is, there is a standard introduction and explanation of expectations for the 
test. It includes a demonstration of the task together with standard materials that are provided to each 
individual who is assessed. The individual is given practice opportunity before the assessment actually 
takes place, and questions and answers are allowed. During the test activity itself, each laborer must 
perform the entire respirator inspection and donning operation exactly to specification in order to pass, 
because when one's life is at stake there is no room for error. 

The test itself involves 13 steps. Trainees must work through the steps in sequence, must 
perform the operation correctly, and must make the correct decisions. The operation itself is easily 
observable by trained instructors who monitor the testing situation and score the exams. The decision- 
making process of the test is more difficult. To assess decision making, trainees are required to talk their 
way through this examination. That is, the laborers require the trainee to indicate what step they are 
undertaking, and then to explain the process that they are using within that step. For example, one step is 
to examine the head straps or head harness for its integrity. Among the things the trainee has to look for 
are breaks, loss of elasticity and twists, broken or malfunctioning buckles or attachments, and excessively 
worn serrations on the head harness that might permit slippage. As the trainee performs that operation, 
they must explain to the observer what they are doing and seeing in each situation, clearly indicaf'ng 
stretched or damaged areas. The observer must hear the trainee explicitly address each issue and 
document - demonstrate whether there is a break a stretch or some other portion of a strap or harness 
that would render it unusable in the training situation. 

The laborers are beginning to generate this type of performance examination for many of the tasks 
in the trades that they perform. They expect to use performance based assessments both to document the 
competencies of their workers and as a training device. They recognize that when there are Standard 
Operating Procedures (SOP's) that must be followed on every job, then creating a performance test that 
requires one to learn those procedures also can be a very effective training tool. Further, by using the 
peiformance assessment as a training tool, they decrease the cost of doing this type of assessment. 

Though a number of trades are moving toward work sample types of assessment that might be 
termed "authentic assessment", one should not assume that paper and pencil tests are unusable, 
inappropriate or without validity. They are useful, especially in combination with other types of tests. 
Furthermore, even the "authentic" assessments, particularly the type of performance test described above 
have difficulties including insuring that each observer will rate a behavior the same way. To that end, the 
Laborers have designed a training program to teach their instructors to become consistent raters of the 
performance. Perhaps, an even greater concern is the cost of providing work sample performance 
assessments. The Laborers believe that it will require establishing "performance assessment days" at their 
training centers where trainees and working laborers who would like to be certified in that skill can return 
and take the assessment in order to earn the certification. Assessment skill days are expensive, but 
provide a good opportunity for training and assessment to the membership. 
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A. INTRODUCTION 

During the course of this study it became increasingly obvious that in, this area of emerging public 
attention and policy development, a great amount of confusion exists simply due to the lack of a common 
language. Throughout this Volume, as well as Volume II and III, the reader will find the same words or 
terms often being used interchangeably. This is to be expected. It happens in our everyday lives - 
constantly. However, common understandings and common frame of references will be essential as the 
skill standards initiative moves forward and at some level common understanding of terms will become of 
major importance. 

What follows is a set of observations from the study team and select set of examples from 
organizations that have already struggled with the issue of terminology and common language. Even from 
those that have struggled to create a common language that explains, for example, the difference between 
certification and licensure one finds very different interpretations. Some differences are provided. There is 
no attempt to suggest what terminology would be preferable — only to note that it will be important to 
have a wide array of interests agree to the common language as this exercise moves forward. 

B. THE EXAMPLES 

1 . From the review of the literature on occupational analysis and assessment the following terminology 
issues surfaced. 



The definition of the word "standards" is somewhat confusing as reported in the literature. The 
confusion seems to stem not so much from the basic definition, or fundamental idea of the word, as from 
the way that the word is used in context. To be explicit, the term "standard", in every sense, seems to 
answer the question "how good is good enough." However, when one examines the way the term is 
actually used in the literature, five or six different expressions surface as ways to define what "good 
enough" means and/or how "good enough" should be examined. For example, sometimes the term means 
the criteria against which one judges performance, activity and operations, regardless of whether that 
performance is individual performance, performance of task groups, or performance of entire organizations. 
When "standards" defines criteria, often it is expressed as profit margin, speed of production, error rate, 
waste and other similar terms. 

Yet criteria is only one of a number of expressions that indicate standards. Other expressions of 
standards include statements of time, such as in traditional apprenticeship programs where 



► Standards 




accomplishment of journeyman status necessarily combines mastery of skills and knowledge through 
required time periods of work experience in various job practices through which the skills are learned. 

Time is also implied as the standard to judge the value of almost all of the education that 
individuals receive. With a few exceptions (discussed in Chapter VI) education credits and credentials are 
simply not awarded unless time has been measured. 

Sometimes the term expresses skills. However, in discussing skills, the conversation may become 
confused because in some situations the discussion relates to depth of skills and knowledge about a 
particular occupation or subject, while in other situations, the conversation relates to breadth of skills and 
knowledges across entire occupational areas. Confounding the issue of skills, sometimes standards are 
expressed in terms of competencies or measurable characteristics and performances of individual job 
holders. 

Two additional expressions of standards are statements of minimum qualifications and goals. 
Often when standards are expressed as minimum qualifications, they indicate the entry-level normative 
expectations for individuals or for organizations that may be looking to achieve some kind of recognition. 
In these situations, for an individual the standard may be a high school diploma or some other type of 
certification. For an organization, standards as minimum qualifications and goals may be expressed as 
quantifiable characteristics such as the elements processes, and programs that institutions of higher 
education must achieve to earn accreditation through one of the regional accreditation agencies in the 
United States. 

► Certification 

The term "certification" in the literature refers to at least two types of activity. In one situation, 
certification relates to achievement of minimum qualifications for entry into an occupation. Once the 
person becomes certified, he/she has the right to practice or to work in that occupation. Frequently this 
type of certification is based on knowledge and may result in a certificate or diploma. 

A complementary use of the word certification is as it relates to specialty skills. Here certification 
typically occurs as one masters additional, specialized, and refined knowledge and skills. The skills might 
be demonstrated at work or in testing situation in order to earn a certificate, and/or qualify to perform the 
specialty skills at work. 

► Performance Tests 

The term "performance tests" expresses two related but somewhat different concepts. In one 
case, performance test means the type of assessment that Linn and others are calling "authentic" 
assessment or "situational" assessment. That is, a performance test measures an individual's ability to 
complete successfully a task within a context like where that task might be performed at work. 
Performance test in that situation might occur "on the job", at an assessment center, or in a training 
situation where an individual is given a job task, tools, equipment, and other materials and asked to 
complete the task to industry established criteria of time and quality. 
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However, "authentic" or "situational" assessment as a performance test is actually a subset of a 
second use of the term "performance test." Initially, the task performance tests was used on the Uniform 
Guidelines of 1978 to mean any type of individual assessment, particularly in a training situation that an 
employee might undertake either for job selection or for promotion. The term includes paper and pencil 
tests, work samples, assessment center test performance appraisals, and any other type of test. 

► Job Analysis 

A series of terms are used synonymously to describe this activity: job analysis, work analysis and 
occupational analysis. Sometimes, however, the term occupational analysis is used to describe a larger 
effort that includes clustering various jobs or occupations into larger industry groups as categorized through 
several different techniques. 

Another term, "task analysis" is frequently added to the literature. Sometimes authors refer to 
task analysis as job analysis. At other times authors refer to task analysis as the process of considering 
job analysis information - specifically job task information - in order to elicit specific skills, knowledges, 
and attitudes that workers must possess and use to perform successfully their job. Other authors 
elaborate further on that definition by including in the discussion of task analysis, the development of 
competency statements for the skills, knowledge and attitudes that workers must possess and display. 

For this study the following definition was used: Job analysis is the systematic process through 
which one investigates and articulates the tasks, skills, knowledge, attitudes, context and other 
characteristics that describe what a worker does in his/her occupation, together with what other workers 
do and the characteristics of organizational context in which the work occurs. 

► Skills 

The term "skill" basically means th3 ability to do something, but the connotation surrounding the 
term can be confusing. The confusion seems to stem from mixing levels cf analysis and levels of 
operation. For example, sometimes the term, "skill" is synonymous with competence suggesting that 
someone is capable to perform whatever skill or task may be associated with that issue. At the same time, 
skill can mean virtuosity and thus something significantly more elaborate then simple competence. 

A second confusion is how much activity is included in the definition of "skill". Sometimes skill 
means a very discrete unit of behavior such as driving a nail at a 90 degree angle to the surface into which 
it's being driven. At other times, skill may mean an entire set of performances such as decision-making 
where a number of factors other skills if you will - are involved in successful performance of the activity. 

Competencies and competency as terms have similar potential confusing elements. Sometimes the 
distinction is simply made between competent and incompetent and expressed in gross measure. At other 
times, competency means an entire cluster of skills and knowledge that a worker possesses and uses. 
Still, a further use of the term is the expression of competence as a single measurable behavior of an 
individual as it relates to a specific knowledge skill or ability or some combination of necessary knowledge, 
skills, and ability in order io perform a task. Whichever level of analysis is used, the underlying notion of 
competency, however, seems to be measurement of successful performance. 
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2. From the review of trade associations, professional societies and others concerned with credentialling 
the following selections of direct quotes ~ some without attribution ~ are provided. 

"Licensing is a mandatory program, usually involving grandfatherism and no real test of 
competence." 

Many of the trade associations do, in fact, have a grandfather clause built into the certification 
system, at least in the beginning years. 

"There is a distinction between certification and registration. Simply stated, AXXX'S 
Certification is official recognition by one's colleagues and peers —it is a specialty 
certification. It is not a substitute for registration — Registration is a legal act on the part 
of states to protect the public — it is a procedure that requires licensing." 

Many of the trade associations, in fact, manage registries as a part of the certification program. 
One organization had clearly given a great deal of thought to helping their members understand the 
different terms and their implications. The name of the organization is not provided. 

► Certification 

Although a few state and provincial governments refer to the term "certification" as a level of legal 
recognition for individual practitioners within that state or province, certification is generally defined as a 
voluntary form of recognition of an individual, granted by an organization or agency which is non- 
governmental. This organizational certification recognizes that an individual has met predetermined 
requirements established by the organization. Certification may recognize only minimum competency 
levels, or may recognize advanced levels of accomplishment or proficiency within a profession. 

► Recertification 

Some organizations require certificate holders to keep certificates current through periodic re- 
examination and/or the recording of continuing education units (CEU's). New certificates, or dated stickers 
affixed to certificates, are issued upon successful completion of each examination. This may be a 
condition for certificate renewal. In some cases, states or provinces may require periodic re-examination of 
specialized areas of knowledge, such as codes. 

► Advanced Certification 

In professions where practice has become specialized, certifying organizations have expanded their 
scope to issue special certificated for highly developed disciplines. In the interior design profession, this 
could include, for example, the areas of commercial design, health care, retail or store planning, hospitality, 
historic preservation, and so forth. 



97 



88 



► Certificate Renewal 

The annual recording of a certificate, by the certificate holder, to keep the certificate current for 
legislative and professional purposes is certificate renewal. 

► Accreditation 

Accreditation applies to programs within institutions, rather than to individuals. Generally, 
accreditation is a voluntary form of recognition and is granted by an agency or association to programs or 
organizations that meet established qualifications and education standards, as determined through initial 
and periodic evaluations. 



The specific terminology for licensing (or legal recognition) of a profession varies among 
jurisdictions. Licensing regulates individuals within a profession, such as interior design. The sole basis for 
licensing individuals is the protection of the health, safety and welfare of the public. Licenses are granted 
by a state or provincial government to individuals who have met predetermined qualifications, generally 
including education and experience requirements, and who have successfully completed an examination for 
minimum competency within a profession. Licensing may limit the use of a title associated with the 
profession, as in the case of title act legislation, or, in the case of practice act legislation, may restrict the 
practice of a profession to those individuals who have become licensed under the legislation. 

► Grandfathering 

A regulating jurisdiction may, at its discretion, issue licenses to long-term practitioners, or others, 
who may not meet all license eligibility criteria as set forth in a licensing statute. For example, licensing 
requires examination, and grandfathering provisions may (but do not always) waive the examination 
criterion. This general acceptance, often offered only during a specified period of time, is called 
grandfathering. Practitioners in allied professions may also be grandfathered if the regulatory agency 
determines that professional and educational backgrounds of the allied practitioners conform to the practice 
or title criteria. 

Yet another example of the use of the term standard can be found from the organization that 
influences the state licensure community. "Standard setting — vhe sound development of an examination 
form is the determination of a standard, or minimum passing score, used to compare and interpret the test 
scores. If adequate consideration is not given to how the minimum passing score is determined, the score 
may not be legally defensible." (Council on Licensure, Enforcement and Regulation (CLEAR). 

3. From the review of education's use of terms. 

Definitions for occupational skill standards, competencies, and tasks are often interwoven in the 
terminology used by education for these processes. States belonging to certain consortiums tend to use 
the definitions developed by that particular consortium. For example, those states using the DACUM 
process normally use definitions as identified by that process. 
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One definition, or a derivative of it, used by many groups is that skill standards are work place 
knowledge, skills, and attitudes required for acceptable job performance. The 26 states belonging to V- 
TECS more commonly use this definition or a derivative of this definition. Some other examples of 
definitions include the following: 

► Arizona defines a task as .... an observable work place activity involving one or more steps, with 
a standard of acceptable performance that can be described and assessed. The state defines 
competency in the following manner: ... an educational "construct" or "concept," an abstraction 
derived from a work place knowledge, skill, or attitude requirement. Acceptable performance is not 
always directly observable, and assessment of competence may require demonstration of 
knowledge in classroom settings. The activities or "competency indicator" to be performed must 
be defined for developing instruction and assessment of competence. 

► Michigan identifies skill standards as job-specific skills which encompass the proficiency to 
perform tasks and technical functions required in occupations. 

► The District of Columbia describes occupational skills as those acquired by observation, study or 
experience in mental or physical performance basic to mastery of school work or other activities. 
Most states do not identify a definition for skill standards, but rather interpret these as 

synonymous with competencies. Standards are interpreted as the measurement by which competencies 
are judged to be successfully completed at a prescribed performance level. 

Dr. Charles Losh, 3 well-known occupational curriculum authority, defines task as an observable 
work place activity involving one or more steps, with a standard of acceptable performance that can be 
described and assessed. Many states subscribe to this definition of task while others have a slightly 
modified version. For example, Michigan and Iowa define task as a unit of work with one or more steps 
and a definite beginning and end. The unit is both measurable and observable. 

A key overarching finding of the review of the state's efforts is that no one set of definitions are 
broadly used throughout all fifty states regarding such terms as: skill, tasks, standard, competency, and 
the term as well as use of assessments have widely different meanings throughout the states. 

The National Governor's Association, recently surveyed state officials regarding the development 
of industry driven skill standard systems. 

State officials surveyed frequently identified the need for clear, commonly agreed upon definitions 
of key terms as a part of an overall framework to guide the development of national, state and industry 
initiatives to develop skill standards. The following terms were most frequently used with varying meaning 
and understanding. 

-* skills -* tasks -> core skills 

— foundation skills -* unit of work — occupational skills 

-+ competencies -* duty -* occupational clusters 

-* skill standards 

This array of examples are not presented to criticize anyone -- simply to point out that it will be 
necessary to pay attention to the development of a common language. 
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CHAPTER VI 
THE QUALITY ASSURANCE PARTNERS 

A. INTRODUCTION 

The next two chapters and the companion Volumes !l and III focus on how skill standards are 
developed and used by both education institutions and industry and occupation based associations. It 
became obvious that, just as it was necessary to provide a back drop about job analysis and assessment 
methods, it would also be a value to provide further information on the organizational infrastructure that 
supports the voluntary system of credentialling in the United States. 

America has placed a heavy reliance upon a network of predominately voluntary organizations to 
promote self-regulation of the education and certification systems. This chapter provides an overview of 
some of the key umbrella organizations that accredit the accreditors or certify the certifiers, or bestow a 
seal of approval on testing and assessment instruments. Adherence to the criteria promulgated by any of 
the following organization's rules and standards are strictly voluntary on the part of an individual institution 
or organization. However, their influence over the accreditation and certification systems are strong and 
their experiences and expertise represent a rich foundation that can contribute to the development of a 
voluntary skill standards system for the country. 

It is important to note that no one organization described below "controls the market" in the 
accreditation, test approval or certification programs. Most of the attention of the organizations that are 
described below focus on the post high school period of preparation and certification of skills. 

B. THE EDUCATION AND ACCREDITATION OVERSEERS 

The American Council on Education, composed of 1500 institutions of higher learning and national 
and regional education associations is an umbrella organization of organizations and focuses on all aspects 
of post secondary education. It operates through a variety of voluntary Commissions and Centers. It's 
Commission on Educational Credit and Credentials acts as part of the self-regulation network that guides 
the operations of the education community in this country. It is the Council's agent on all matters relating 
to educational credit. The Commission has the task of assuring the integrity of academic credit and 
credentials. It works to improve credit and credentialing systems, as well as the quality of information that 
credit and credentials convey for both education and social purposes. It also assists public policy makers in 
developing and establishing policies that foster adult learning. 
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The chief functions of the Center for Adult Learning and Educational Credentials, (the operating 
arm of the Commission) include: 

► Assisting institutions and organizations in developing policies and procedures for formally 
recognizing extrainstitutional learning, integrating this learning into students' program of study, and 
applying it to credential requirements. 

► Operating programs that assess extratinstitutional learning including the a) General Educational 
Development (GED) Testing Service, b) Credit by Examination Program (CBE), c) Military 
Evaluations Program, d) Program on Noncollegiate Sponsored Instruction (PONSI), and e) ACE 
Registries of Credit Recommendations. 

An example of one of the programs in PONSI that builds upon the work of Center is to provide 
college recognition for formal military education and training. Currently the PONSI course assessment 
process allows trade associations, apprenticeship training programs, individual businesses' training 
programs as well as the military to have courses approved for college credit. Credit is recommended for 
vocational certificates through graduate studies. It is at the discretion of individual education institutions to 
accept this form of credit transfer. 

The Credit by Examination program is designed for an individual to skip the requirement of taking 
selected courses by "testing out." Again, it is at the discretion of individual education institutions to 
accept the examination in lieu of taking the course. 

The Council collaborates with the Council for Adult and Experiential Learning (CAEL), an 
independent organization, that promotes recognition of prior learning assessment (PLA) gained outside the 
classroom. In 1991 CAEL conducted a survey of all postsecondary institutions accredited by agencies 
recognized by the Council on Postsecondary Accreditation (COPA), described below, to assess the status 
of PLA practices. 

For the purpose of the survey PLA was defined broadly to include ail modes of assessing learning 
gained outside the classroom. It included standardized examinations, including Advanced Placement (AP), 
College Level Placement Examination Program (CLEP), the American College Testing's Proficiency 
Examination Program (ACT/PEP) and the Defense Activity for Non-Traditional Education Support (DANTES) 
examination. It also included course challenge or departmental examinations including ACE's Military and 
PONSI and individualized programs that focus on portfolio-assisted assessments, oral interview and 
competence demonstrations. 

The survey had a 47% response rate (1,732 institutions) and the key finding is that 86% of 
responding institutions report that they assess prior learning through standardized examinations, course 
challenge examinations, ACE recommendations, or portfolio assessment. Again, freedom reigns in that 
each institution can select the type of assessment it will accept and for what purpose. There are other 
findings of note in the study that need not be described here. What is important for this discussion is that 
there is a growth of institutions willing to accept some form of documentation of learning gained outside of 
the classroom if there has been some type of quality assurance like screening of the assessment 
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instruments by some established and respected third party organization (Fugate and Chapman: 1992). 

The Council on Postsecondary Accreditation serves as an umbrella standards setting organization 
for the voluntary act of an education institution to seek either institutional recognition or to have programs 
offered within the institution to be recognized by a specialized accrediting body. Accreditation is defined 
by COPA as: 

M .. a communal, self-regulatory process by which nongovernmental voluntary associations 
recognize educational institutions or programs that have been found to meet or exceed stated 
criteria of educational quality and assist in the further improvement of the institutions or 
programs." 

How voluntary is voluntary is open to question. Since 1952 the federal government has tied 
accreditation to eligibility for federal funds for student grant and loan programs. The result is that any 
educational institution that wants to have a substantial student body will seek accreditation. The 
Department of Education now has the responsibility to recognize accrediting agencies and it is through this 
recognition that an individual school is determined to be eligible to obtain federal funds such as federal job 
training monies and its students determined to be eligible to apply for federal grants and loans. 

There are regional and national institutional accrediting bodies recognized by COPA and the U.S. 
Department of Education. The national bodies accredit particular types of institutions such as the private 
proprietary schools. 

Institutional accreditation is not automatically conferred upon all of the programs within an 
accredited institution. There are a large number of specialized accrediting organizations recognized by 
either or both COPA and the U.S. Department of Education. These specialized accreditation organizations 
focus on specific occupations such as architecture, business administration, construction, journalism and 
others. Additionally there are several specialized accreditation organizations that are intricately linked to 
either certification or licensure such as the health and allied health fields, engineering, forestry, landscape 
architecture, social work, and teacher education. (Lenn: 1987) 

Often industry trade associations or organizations focused on professional occupations are key 
players in the accreditation of education programs, indeed often the initiator for creating an accreditation 
program. There is some tension within the accreditation community regarding the seeming proliferation of 
specialized accreditation bodies. The argument follows the line of reasoning that institutional flexibility is 
hampered and cost are increased when a second tier assessment process necessitated by the specialized 
accreditation is added onto institutional accreditation. This report was not meant to review in depth the 
issues of accreditation but it can be expected that, as the skills standard system evolve, there will be a 
need to address the interrelationships between specialty accreditation and the skill standards system. 

This is a period of some turmoil within the accreditation community as it tries to sort out the 
appropriate role of the accreditation body in the whole arena of measuring institutional effectiveness. The 
tradition has been that accreditation requirements have focused on the adequacy of facility, faculty, 
curriculum, and other "input" criteria. Yet the increased call of policy makers to focus on an array of 
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outcome criteria to measure institutional and program effectiveness has helped to fuel a substantial debate 
regarding the appropriate role of these voluntary promoters of self-assessment to promote quality. Some 
of the specialized accreditation bodies have adopted the criteria that an institution must have some form of 
measuring outcome results such as showing how many of their students were able to pass a nationally 
recognized certification or state licensing exam. It is not so easy for the institutional accreditation bodies 
to demand such specific proof of results and the search for the correct balance continues. 

Public schools are also accredited, often by regional accrediting bodies, however, the states 
maintain substantial legislative oversight over this process. 

C. CERTIFIERS OVERSEERS 

The American National Standards Institute (ANSI) was founded in 1918. It is an umbrella 
organization that does not establish standards; this task is done through 250 qualified technical and 
professional societies, trade associations and 1,100 individual companies that voluntarily submit them to 
ANSI for approval. Over 9500 standards have been approved by the organization, mostly for products. It 
has just recently revised its criteria for approval to assist U.S. organizations be better positioned for 
international trade as a result of several international trade agreements. 

ANSI is the U.S. member body of the International Organization for Standards (ISO.) All of the 
other countries are represented by the central governments. ISO standards, while voluntary, are 
increasingly being used to assure conformance of products and services across international boundaries. 
The ISO 9000 series is a set of technical documents that set forth criteria focused on quality assurances. 
Embedded in these standards are criteria relating to the quality of the workforce. The standards focus on 
quality assurances including documentation of processes to determine the kind of skills needed to produce 
a given product. Companies wanting to participate in the European market are increasingly being required 
to show proof that workers have the appropriate credentials to perform their jobs. 

ANSI's criteria for approving any accreditation program of an individual company, association or 
institution developing and maintaining certification and accreditation programs includes (but is not limited 
to) the entities': 

► Being operated by a certification body that has acknowledged and/or defensible independent 
third-party status as a certifier of products, processes, or services; 

► Having clear procedural guides that provide details of the parties involved in the program; 

► Making "full disclosure" of information concerning the accreditation activity; and, 

► Subjecting their program to an annual review by ANSI. 

This short list obviously does not capture all of the procedures and quality assurance criteria that 
ANSI has developed, including the development of a legal contract between ANSI and the certifying 
organization. Any organization developing standards using the ANSI process must agree to review the 
standards that have been agreed to by the industry on a regular basis (most of the skill standards must be 
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reviewed at least every five years.) There are several levels of agreements between ANSI and organizations 
that establish standards, many of which do not warrant the seal of "accreditation" or the highest level of 
recognition by the organization. 

An example of an organization that is using the ANSI processes to develop national voluntary 
standards is the American Society for Quality Control (ASQC). ASQC has been developing guidelines for 
using quality assurance principles in education and training programs. The draft guidelines have been 
developed to be technically equivalent to the relevant ISO standards. For those organizations concerned 
with the establishment of world class benchmarks for either the instructional process or documentation of 
skills and knowledge required to prove employees are able to meet international standards the ASQC 
materials are worthy of review. 

A second overseer, the National Commission for Certifying Agencies (NCCA) was created in 1989. 
It is an organization of organizations established by the National Organization for Competency Assurance 
(NOCA) as a commission to establish national voluntary standards for and recognize compliance with these 
standards by agencies certifying individuals in a wide range of professions and occupations. NCCA 
approval criteria and standards are the only national and voluntary standards for certification agencies, in 
essence it certifies the certifiers. 

This new Commission's predecessor was a similar organization that only focused on the health 
certification agencies. It was originally supported by the federal government (Dept. of Health and Human 
Services) to serve as a platform for the development of standards of excellence in private certification. 

The parent organization, NOCA, is a membership association whose mission is to provide a forum 
for all types of practitioners and organizations interested in competency assurance and certification. 
Members include employers, state licensing boards, educators, federal agencies, test developers, consumer 
groups, certifying agencies and professional associations. 

Underlying the NCCA's effort is the concept of "competence" which, according to NCCA, has yet 
to become a term that is acceptable to all who use it. For NCCA the emphasis of competence is on 
results. Competence means: work is to be performed accurately, correct judgments are made and 
interactions are conducted in a professional manner, it must be demonstrated and maintained throughout 
ones' practicing life. 

The following eligibility requirements must be met before a certification program can even apply for 
review by NCCA. The program: 

1. Shall be non-governmental, unless the certification is for government employees. 

2. Must be national in scope. 

3. Must be operated by a not-for -profit agency. 

4. Shall have completed at least two national examination administrations prior to seeking NCCA 
approval. 

The program must be administratively independent and this means that all policy decisions relating 
to certification matters are the sole decision of the certifying body and not subject to approval by any other 
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body and that ail financial matters related to the operation of the certifying component are segregated from 
those of the professional association. Other portions of the guidelines require that the certifying body of a 
professional organization shall be separate from the accrediting body of the professional association. 

There must be alternative pathways for individuals seeking certification. This means individuals 
cannot be required to have attended a specific type of accredited education institution, be required to take 
the specific courses only offered by a sponsoring organization, nor have had to complete a specified 
educational degree. 

The guidelines require that certifying boards meet a series of psychometric and policy standards 
such as the Standards for Educational and Psychological Testing and the Uniform Guidelines for Employee 
Selection Procedures. NCCA, through a joint endeavor with the Council on Licensure Enforcement and 
Regulation (CLEAR), is in the process of developing an Examining Guide for Credentialing Boards based on 
the Principles of Fairness to supplement the aforementioned guidelines used by the courts and the federal 
government. 

CLEAR is another actor in the national quality assurance network. It is an affiliate organization of 
the Council of State Governments and is a voluntary organization of state licensure and testing bodies. 

D. REVIEWER OF TESTS 

There is an organization that has recently been established to attest to the quality of tests in the 
public sector. The National Public Service Accreditation Board (NPSAB), established through the auspices 
of ICMA, a professional association of appointed administrators serving local governments. ICMA 
responded to the concerns of local government officials regarding the efficacy of a wide range of tests to 
credential, select and promote public service employees. The current board members of the Accreditation 
Board consist of representatives of the National League of Cities, the Public Risk Management Association, 
the National Architectural Registration Boards, and the Fraternal Order of Police, plus an academic expert in 
the field of educational measurement. 

The cornerstone of this Board's effort is to accredit tests and its approval is program specific with 
the exception of certain general aptitude or selection tests. Different tests for the same occupation may be 
accredited but accreditation of tests is not automatically transferred between users due to the concern that 
a specific test may not be appropriate to a specific program. 

NPSAB maintains a listing of all accredited tests and sponsoring organizations. Any organization 
providing tests to a state or local government (such as professional associations, industry associations, 
municipal leagues, employee unions, private testing firms) may apply for accreditation for each of the tests 
they provide or the local unit of government may submit the tests they are considering or have developed 
in-house. 
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E. THE FEDERAL ROLE 

The federal government performs several critical roles in the standard setting/quality assurance 
systems in the U.S. One of these is publishing the National Institute of Standards and Technology's (NIST) 
Standards Activities of Organizations in the United States. The latest edition was published in 1991, the 
first in 1941. It includes a wealth of information including some related to credentialing/standard setting 
bodies. 

OMB circular A-119, titled Federal Participation in the Development and use of Voluntary 
Standards, issued in 1982 provides guidance to federal agencies in this area. There is an interagency 
committee on standards policy established in 1968, however, it appears that most of the federal 
governments attention has not focused on the issues concerning credentialing programs. 

The Federal Trade Commission and the Department of Justice have scrutinized professional 
credentialing programs. Courts have been most concerned about credentialing when it generates 
significant economic benefits to those that have the credential and when it denies those without the ability 
to practice in the profession. 

A review of court decisions suggests that the following reasonable criteria be applied in the 
development of a credentialing program. 

► Criteria should be no more stringent than necessary to assure minimum qualifications, especially 
where credentialing is of significant economic value. 

► Criteria must not have the purpose or effect of unreasonably restricting or boycotting 
competitors. 

► Criteria should be established only after reasonable notice and opportunity to participate is 
afforded to all those who may be affected by credentialing requirements, including potential 
candidates and users of their services. 

In addition, procedures used in the credentialing program have been reviewed by the courts to 
ascertain if they are objective and impartial. The following guidelines, if adhered to, apparently can 
withstand legal traps. 

► Participation should ordinarily be voluntary and open to those who are not members of the 
association. 

► All candidates should be treated equally; it may not be considered fair to exempt current 
association members from new credentialing requirements. 

► While general promotion of a credentialing program is not a problem, associations should not 
promote credentialed individuals by name or disparage the noncredentialed. 

► Credentialing should not be used to "blackball" or limit the number of competitors arbitrarily. 

► Denial of credentials should be made by written notice, giving the reasons for denial; opportunity 
for an appeal in writing or at a hearing should be offered, to be decided by a body other than the 
one which made the initial decision. 
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► Decisions on applications should be made by an objective body not composed exclusively of 
credentialed individuals who might stand to gain financially from a decision affecting competitors. 
(LAD, 1992). 

As can be seen from this cursory review there exists a complex array of voluntary and government 
entities that are already actively involved in the quality assurance of education and certification systems. H 
will, no doubt, be necessary to involve these and other organizations in the design and development of the 
voluntary skill standards network. 
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RESOURCE LIST FOR ORGANIZATIONS DISCUSSED IN THIS CHAPTER 



American Council on Education, The Center for Adult Learning and Educational Credentials, 1 Dupont 
Circle, NW Washington, DC 20036-1193 

American National Standards Institute, 
1430 Broadway New York, N.Y. 

American Society for Quality Control 

310 West Wisconsin Avenue, Milwaukee, Wisconsin 53203 

Council on Licensure, Enforcement and Regulation, 

3560 Iron Works Pike, P.O. Box 11910, Lexington, Ky. 40578-1910, 

Council on Postsecondary Accreditation, 

1 Dupont Circle, suite 305, Washington D.C. 20036 

National Organization for Competency Assurance, 

1101 Connecticut Ave. N.W. Suite 7u0, Washington D.C. 20036 

(NOCA has developed a useful guide, titled Starting a Certification Program, that can be ordered at the 
above address.) 

National Public Service Accreditation Board, 

Suite 500, 777 N Capitol St., NE, Washington, D.C. 20002-4201 
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CHAPTER VII 

DEVELOPMENT OF OCCUPATIONAL SKILL STANDARDS IN EDUCATION 



9 

ERIC 



A. INTRODUCTION 

Today's structure of occupational skill standards is a reflection of the past 20 years of change in 
business and industry. Over time the effects of industrial change pushed forward a process of change in 
the skills required by industry, and in turn this process brought changes to skill standards setting. 

Starting in the seventies, and picking up speed in the eighties a new competency based system 
emerged. Through the joint efforts of employers and education, this system of competencies, duties, and 
task lists has become ever more sophisticated in the eighties. 

The relationship between education and employers in the development of skill standards started at 
various points in the states. Four states initiated industry related skill standards prior to 1970. Wisconsin 
first established occupational skill standards in 1906, with the beginning of their vocational education 
system. During the 1970s, 11 states moved into business and industry partnerships to develop 
occupational skill standards and related materials. In the 1980s, 21 states moved into the occupational 
skill standards system. 

The first federal act which addressed the development of skill standards was the Carl D. Perkins 
Act of 1984. This act required at least two state level technical committees to be established for the 
purpose of setting skill standards for specific occupational clusters. This enhanced the movement that was 
already underway. 

Many states developed more than the two required committees. Each year, as education found 
success in establishing skill standards with business and industry, they gained willing partners. Business 
partners saw the need for equipment, for instructor retraining, and for transition-to-work programs in the 
form of expanded cooperation in business, and youth apprenticeship. They also saw the need to 
strengthen the basic financial structure and became the crusaders for the work force program. 

Occupational skill standards have been established primarily through the efforts of state levels of 
vocational-technical education. Although many local education agencies at the secondary and 
postsecondary levels develop occupational skill lists with business and industry, they are usually developed 
for a specific community or a small geographic area. 

Due to the 1990 reauthorization of the vocational education act, popularly called Perkins II, new 
opportunities are now available to push forward the utilization of occupational skill standards within the 
states. This legislation, for the first time, provides fiscal support for national occupational standards to be 
developed. Developmental work is currently underway to establish a larger pool of national industry- 
recognized standards. The U.S. Department of Education and the U.S. Department of Labor, in concert 
with national business and industry trade associations are taking the lead. This effort has just commenced 
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and will continue for one to three years, depending on the availability of funds and the success of individual 
projects. 

States also have the opportunity to develop performance standards for the vocational programs in 
their states due to the new Perkins II legislation and the chance to improve the articulation process 
between secondary and postsecondary education institutions through the Technical Preparation programs. 
As will be seen in the states' responses to the expansion of occupational skills standards, a melding is 
occuring between these strands, and others. 

The existing state systems have been supported by the National Curriculum Network with centers 
established in each region of the country. The Network's purpose is to disseminate standards, curriculum, 
and instructional materials for vocational-technical education. Several other curriculum consortiums also 
have been instrumental in providing curriculum for existing systems. The Mid-America Vocational 
Curriculum Consortium (MAVCC) is an example of a consortium of states that develops occupational skill 
standards and curriculum as a joint effort of 14 states. 

Several other consortiums of states address specialized areas of the vocational-technical education 
curriculum, such as entrepreneurship, marketing, and home economics. 

Throughout this chapter, for purposes of clarification, occupational skill standards will be used as 
synonymous with competencies. Tasks will be referred to as components of competencies or occupational 
skill standards. 

B. THE DEVELOPMENT OF OCCUPATIONAL STANDARDS IN THE STATES 

What follows is a summary of the activities taking place in all of the states relating to the 
development and use of occupational skill standards initiatives. Over 700 separate industry education 
committees were identified throughout the country being actively involved in the development of skills 
standards. The source of the material from the states that is analyzed is based on a study conducted by 
the National Vocational-Technical Education Foundation. The survey instrument is an Appendix in Volume 
II. 

The Drivers 

The educational development of occupational sxill standards has been driven by the specific needs 
of industry. It is not an easy task to be responsive to the needs of businesses and industries during a time 
when there are so many shifts taking place within the total economy. However, the vocational education 
officials within the states have been in the lead to reach out to industry to establish the frameworks within 
which such standard setting exercises can occur. 

The state officials, of course, must work within the context of the governance and total education 
infrastructure that exists within their particular state. Such contextual realities have not always made it 
easy to respond to the needs of industry and business as directly as some leaders would like. 
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The "bottom line" is that industry and business need employees with skills to do a specific cluster 
of jobs. Along with these job specific skills, employability, related academic skills, and SCANS skills also 
are needed. Business and industry use all of these skills requirements to hire, upgrade, and promote 
employees. 

Vocational-technical education's primary approach has been to use occupational skill standards to 
develop task lists as the basic units of curriculum, instruction, and evaluation criteria to ensure that the 
students acquire the needed skills for jobs with business and industry. After the Carl D. Perkins Act of 
1990, at least 33 states have incorporated occupational skill standards as a criteria for vocational-technical 
education performance standards. This helps ensure that support structures will be in place to attain the 
program outcomes required by industry. Specific purposes for development of occupational skill standards 
by states include the following: 

1 . Forty-eight states indicated occupational skill standards were established for the purpose of 
curriculum development; 

2. Forty-six states indicated, also, that occupational skill standards and competencies were 
established as guidelines for courses and programs; 

3. Twenty-nine states indicated skill standards were established for the purpose of developing 
criteria for assessing student mastery; 

4. Mentioned also by at least one state were: vocational student organization competitive events; 
instructional materials development; program approval; program completion; development of 
minimum competencies, and others. 

The Initiators 

The development of occupational skill standards most frequently has been initiated by state 
agencies or consortiums; but in all cases there has been an important involvement of industry: 

1. In only six states did business, industry, and education jointly organize the occupational skill 
standards process; 

2. One state, New Hampshire, credits industry and business alone for originating the effort; and 

3. The remaining states credit education as the originator. 

It is well known that states have, since the mid 1980s, been enacting laws requiring all levels and 
areas of education to move to measurable and accountable standards. Occupational skill standards are 
clearly a part of that response. Neither the employers nor those who most needed skilled job training have 
been satisfied with the education being provided. A massive upgrading of technical job preparation 
programs is finally being recognized as a key response to these rapid economic and industrial shifts. 

Additionally, equity legislation has changed the occupations now open to males and females. It 
once was uncommon for a man to become a nurse. Today, many men have entered this career. Similarly, 
today women are. accepted in many trades. Education is combatting the problem of gender discrimination 
in vocational-technical programs in a variety of ways. These include: 

in 
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1 . State occupational skill standards are being examined for gender bias in 37 states; 

2. Twelve states directly involve the sex equity coordinator in the development of ail occupational 
skill standards and related curriculum; and 

3. Sex equity coordinators in 10 additional states review all occupational skill standards and 
curriculum for biases. 

Influence of Skills Standards on Course of Study 

A primary use of occupational skill standards is in the development of programs and courses of 
study at the local, secondary, and postsecondary district level. State vocational-technical education 
agencies and divisions indicate the following: 

1. Forty-seven states use occupational skill standards as a basis of articulation between secondary 
and postsecondary programs and courses; 

2. Forty-four states indicate standards are the basis for the development of syllabi for courses; 

3. Thirty-six states indicate skill standards and task lists are the basis for program certificates of 
mastery at the postsecondary level; and 

4. Forty-two states use occupational skill standards as a basis for testing and assessing skills and 
tasks acquired. 

The Mid-America Vocational Curriculum Consortium's major focus is on the development of 
curriculum and it brings business, industry and education together as a committee to establish the 
occupational skill standards lists, according to Dr. Jim Steward, Executive Director. Both educators and 
business representatives from each state serve on development teams. Both groups are involved in the 
validation processes. 

MAVCC materials are written for occupations which member states are not developing. If a well- 
developed curriculum based on validated occupational skill standards is available, MAVCC does not 
duplicate the effort. 

Partnerships 

Partnerships are built on the needs of all those involved. Business and industry are accustomed to 
being in touch with their suppliers, and for many occupations vocational-technical education is industry's 
supplier. 

In the development of the occupational skill standards, education sought the help of their business 
and industry partners, with the following results: 

1 . Forty-one states called upon managers from business and industry to help form the 
occupational skill standards committee. 

2. Forty-three states actually used incumbent workers or supervisors who had worked for at least 
two years on the job being studied. 
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3. Forty states used instructors from the occupational field involved with the partnership, and 34 
states used vocational education administrators in addition to instructors. 

4. Twenty-eight states introduced a new partner to this development, the academic instructor. 

5. Thirty-five states now use curriculum developers to work with the occupational skill standards 
process. 

As noted in Chapter III many states have placed heavy reliance upon partnerships with one another 
in the development of job analysis materials that are then turned into curricula. The V-TECS consortium has 
been funded through the states and with both state and federal funds. Currently, V-TECS annual 
membership fee is $20,000 and in addition each member state must commit to the development of two 
products per year that are then shared with all of the other states. These products could be the 
development of test bank items, duty task lists and could cost the state an additional $5,000 to $10,000 a 
year. This shared cost approach has helped assure the return is significantly greater than independent state 
work could be. As lessons are learned over the years V-TECS has tightened up both the process and the 
format in order to help assure common language and understandings between industry and education. 

This partnership is managed by each member having the right to appoint one member, to the 
V-TECH Board, normally it is the director of the Vocational-Technical Education program in the state. 
Additionally, ail of the military services are associate members. 

Financing the Development of the System 

During the 1970s much of the state development of occupational skills and competencies was 
financed by Federal vocational education funds (Table 6). States indicate the following: 

1 . Forty-nine states used federal vocational education funds for development of the 
occupational skills and competency lists; 

2. Thirty-four states used state funds in addition to federal funds; and 

3. Only sixteen states used business and industry funds for development of standards. 
Estimates of costs to develop one set of occupational skill and competency standards varies widely 

among states. Costs vary greatly from one occupational cluster to another. Some are more complex and 
have a larger number of jobs. The highest costs for development of one occupational skill standard area 
were reported by Florida and Utah. Florida spent $289,000 for the establishment of an expansive health 
occupations cluster, while Utah spent $504,000 for the development of an equally large business 
occupations cluster. These costs included industry validated occupational skills and competencies, 
curriculum development, and related instructional materials. Kentucky estimated their effort to establish 
skills and related materials in all occupational areas since they started the program twenty years ago to be 
approximately $1,000,000. States have spent widely varying amounts on occupational skill standards per 
occupational cluster. The range is between $3,000 to $20,000. The great variance in costs may have to 
do with the extent to which materials were developed. When lists alone are developed, less funding is 
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Funding Source 


Percent of States 

Using Each 
Funding Source 


Federal Vocational Education Funds 


94 


Other Federal Education Funds 


12 


State Vocational Education Funds 


65 


Other State Education Funds 


25 


Business/Industry Funds 


31 



Sote: State responses indicated the use of multiple sources of funding for developmenet of occupational skill standards. 

required than when curriculum and associated instructional materials are developed. Also, some costs vary 
in the development of the standards lists. If a structured group interview process is conducted in various 
parts of a state, the costs may be higher than when fewer people and validation processes involved. 

Approximately three quarters of the states are committed to funding on-going maintenance and 
revision of occupational skill standards and related competencies and tasks lists. However, actual 
maintenance and revisions may be accomplished by a local secondar", postsecondary, or higher education 
institution funded by the state to perform the work for the state. Fourteen states indicate that skill 
standards development is a local district issue, reflecting the overarching educational traditions in those 
states. 

The cost may have nothing to do with the quality or validity of the occupational skill standards, but 
may have to do with the number of people involved and the quantity of materials prepared in addition to 
the actual standards lists. 

Extent of Coverage 

The National Occupational Information Coordination Council (NOICC) and its state counterpart, 
State Occupational Information Coordination Council (SOICC), have provided organized categories of job 
classifications that have been heavily used in the development of the systems. This organized set of 
occupations provides data on the types of jobs in demand within a state, the jobs considered to be part of 
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an occupational cluster, and basic descriptive information concerning an occupational cluster. They also 
use this information to determine for which jobs there is the greatest demand, which require more training 
than others, which are new positions, etc. 

This information is supplemented by other information bases including The National Curriculum 
Network; Consortiums, such as V-TECS and MAVCC; The National Center for Vocational-Technical 
Education; The Ohio State University Center on Employment and Training; state curriculum centers; trade 
associations involved in the National Skill Standards projects; and other trade associations. Examples of 
these include: National Automotive Technician Education Foundation (NATEF), American Hotel and Motel 
Association, Education Institute {AHMA-EI), and Electronics Industries Foundation. 

V-TECS has developed occupational skill standards, which they call catalogs, with duty and task 
lists identified for approximately 110 occupations. The occupational skill standards and related technical 
committees developed by education at the state level include 61 occupations within content areas of 
agriculture, business, marketing, health occupations, trade and technical occupations, personal services, 
and home economics. 

The following tables, Tables 7-15, provide the summary of activity in each of these occupational 

clusters. 

SKILL STANDARDS BY 
OCCUPATIONAL CLUSTER 



Table #7 
Agriculture 





Horticulture 


Landscape 


Production 
& Mgmt. 


RancrartQ 
Fanning 


Mechanics 


Animal 
Science 


Natural 

Resources 

Forestry 


General 


Total 


Number of 
Occupational 
Sk« Lists 


16 


15 


15 


13 


13 


13 


18 


18 


120 



Table #8 
Automotive Technology 





Auto Body 
Repair 


Auto Mechanic. 
Dotal, Parts 
ft Other 


Diesel (Heavy 
Equip. Mech.) 


Recreational 
Marine 


General 


Total 


Number of 
Occupational 
Stii Lists 


16 


16 


13 


12 


9 


66 



9 

ERIC 



115 



106 

Table #9 
Business 





Computsr 
(Wordprocassing 
* Othar) 


Claries! 


AMi. 
Assistant 


Data 


Socratarisf 
(Oar., Esse. 
Admin. Lags!) 


Non-Dagras 

Acctg 


aSd- 

MyRIL 


Ganaral 


Totsi 


Numbar off 
Occupational 
Skal Lists 


18 


16 


16 


15 


16 


14 


13 


32 


140 



Table #10 
Health Occupations 





Technician* (EMT, 
Racaology, Othar) 


Nursing (RN, LPN, 
Asst/Aida) 


DsntalfAsst 
Tach, Othar) 


Ganaral 


Total 


Numbar off 
Occupational 
Ski Lists 


12 


17 


12 


21 


62 



Table #11 
Manufacturing Technology 





Drafting 
CAD/CA 
M 


Shaat Matsl 
Machinist 


Pacti until 


Industrial 
Makitananca 


Appianca 
Rapak 


Ganaral 


Total 


Numbar off 
Occupational 

SU Lists 


16 


9 


14 


11 


4 


10 


64 


Table #12 
Marketing 






Salas 


Mark sting 


Entrapranaurstup 


Ganaral 


Total 




Numbar of 
Occupational 
SUM Lists 


8 


11 


9 


16 


44 
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There are separate stories to be told behind each of the numbers and categories listed in the 
Tables. Issues such as licensure requirements play a heavy role in some clusters, arrangements have been 
developed with apprenticeship programs to use their skill standards in several of the constructions 
occupations, manufacturing firms have helped to develop new CAD/CAM standards, individual firms 
commit their own resources to train the instructors and test students, and the list continues. 

Each year updating is required in some areas. The most difficult occupations to keep updated are 
those with changing technologies such as printing, electronics, manufacturing, automotive, and specialty 
areas in health technologies. Keeping these expansive occupational skill standards lists up-to-date is one of 
the greatest concerns of education and its partners. 

Although occupational skill standards and related matenals are being developed by states, 
consortiums of states and by apprenticeship groups, no one set of skill standards has been established for 
ail states or is used in every state. Extensive lists of occupational skill standards are available across the 
nation. However, only 26 to 32 states use a common set of standards for any one occupation. Overlaps 
may be found in some criteria as a result of a diversified system of occupational skill standards across the 
United States. 

The Standards Setting and Occupational Analysis Process 

Occupational skill standards are being established at the state level using representatives from 
industry; incumbent workers and their supervisors; local education administrators and appropriate 
occupational instructors; curriculum developers, and researchers. 

As was discussed in Chapter ill validation of the occupational requirements is essential. 

1. Thirty-eight states start with V-TECS catalogs to compile a list of existing skills for a selected 
occupation. 

2. Thirty-three states use consortium lists and materials. Among consortiums used to develop a 
basic list were the following: Mid "America Vocational Curriculum Consortium (MAVCC), 
Individualized Distributive Education Curriculum Consortium (IDECC), National Network for 
Computing Curriculum for Vocational Technical Education (NNCCVTE), American Agriculture 
Association for Vocational Instructional Materials (AAVIM), East Central Network for Curriculum 
Coordination (ECNCC), and others. 

3. Thirty-seven states use lists developed by business and industry as a starting point. Arkansas 
and Oklahoma most frequently use business lists as their starting points. 

4. Thirty states use previous state and local standards lists as starting points for developing new 
standards. South Dakota, North Dakota, New Mexico, Delaware, and Pennsylvania use their 
previous state lists while Utah, Florida, New York, Ohio, and Wyoming use combinations of other 
states' lists, as well as their own. Maryland uses previous DACUM lists. 

5. Twenty-one states use apprenticeship lists as a starting point for skill standard development. 

o 
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Figure 6 details the sources of materials used to establish occupational skills standards. 



FIGURE 6 SOURCES OF MATERIALS USED TO ESTABLISH OCCUPATIONAL SKILL 
STANDARDS 
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FIGURE 7 PROCESSES USED TO ESTABLISH STATE LEVEL OCCUPATIONAL SKILL 
STANDARDS LISTS 
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In establishing occupational skill standards, most of the states, as seen in Figure 7, use a process 
involving input from workers in jobs for which these standards are being developed. These processes 
include the DACUM or modified DACUM process, the V-TECS or modified V-TECS process, and various 
other processes utilizing workers and others. 

Thirty-four states use DACUM or a Modified DACUM process for the establishment of occu> itional 
skill standards and competencies. Modifications may include starting with validated or invalidated sets of 
skills developed by earlier groups from industry or combinations of education and industry/business. 

Twenty-nine states indicated they use another industry process utilizing workers and others. Some 
examples include the incorporation of occupational analysis and the utilization of input from postsecondary 
instructors. Twenty-six states use the V-TECS or a modified V-TECS process for the development of skills, 
and performance objectives with business and industry. Modified processes may inciude variations by 
states in accordance with already established processes with business/industyr. Fifteen states indicated 
the use of position description or task analysis at the job site as a primary method to develop occupational 
skill standards. 

A labor-apprenticeship process is used by 1 1 states to establish occupational skill standards. A 
system developed by the federal Bureau of Apprenticeship and Training provides skill and task lists which 
are sanctioned by federal and state agencies as a basis for training and certification. In addition, many 
labor unions have certified apprenticeship programs which have labor - and industry - sanctioned task lists 
for the purpose of training and certification. 

A combination of the described processes is used by 20 states to establish their occupational skill 
standards. Virginia uses a curriculum center to prepare tentative task lists from all available sources. A 
modified DACUM process, technical committee, or both are convened to validate or revise tentative lists. 
Then, curriculum is constructed according to the validated occupational skill lists. Vermont conducts 
instructor curriculum round table meetings to define program areas and create rough drafts of competency 
lists. These lists are validated by a state technical committee. 

Some states use the Ohio Competency Analysis Profiles which is an expanded, modified DACUM 
process and committees developed model curricula which subsequently was validated by educators, 
business, industry, and commerce. 

Iowa requires minimum competency lists containing essential occupational skills which lead to 
entry level employment. However, these are not intended to be the only competencies leaded by 
students. Many additional competencies exist within Iowa programs. 

Colorado community college faculties describe the common elements of secondary and 
postsecondary curricula and guarantee articulation, statewide, between these two levels of education. 
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The Assessment Process 

Although assessment is a key component of the occupational skill standards system, less than one 
half of the states have developed and validated test banks at the state level. More than one-half of the 
states indicate that local districts within their boundaries have test bank questions available for instructor 
use (Table 16). 

At least 30 states indicated they do not validate test banks. Arkansas, Florida, Georgia, 
Oklahoma, South Carolina, and Wisconsin are among the states which do have validated test banks 
available. 

TABLE #16 AGENCIES AND GROUPS PROVIDING ASSESSMENT AND CERTIFICATION FOR 
OCCUPATIONAL SKILLS ACQUIRED 



Agencies and Groups 


Percent of States 


Local Education 


58 


State Vocational-Technical Education 


46 


Business/industry 


44 


Labor 


21 



Note- State responses indicated muttipk agency invoh/ement in the assessment and certification of occupational skiH. 

Sources of test bank items other than at state or local levels, include V-TECS Test Banks, 
SOCATS, and NOCTI. 

Of the states that have test item banks one of the most extensive is Oklahoma, which has items 
for occupational skills, related academics, and employability skills. Other states which have developed test 
items or test item banks at the state level include: Arizona, Florida, Michigan, North Carolina, Utah, 
Vermont, and Wisconsin. In addition, Minnesota and New York have local test item banks for secondary 
and postsecondary levels. New Hampshire and Oregon are in the process of creating tests. Florida has 
developed more than 30 test item banks coded to statewide curriculum frameworks. Michigan has a 
microcomputer-based test item bank to provide teachers with test items directly related to the curriculum 
taught in their programs. The program allows the user to randomly select test items for tasks they have 
identified. North Carolina's development of computerized test item banks is approximately 50 percent 
complete. 
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Types of tests or assessments used to determine mastery of occupational skills include written, 
computerized, cognitive, simulation, situation, actual performance, and combinations of tests. 

1 . Forty-one states use written cognitive tests, with only one half of these using computerized 
versions of such tests. 

2. Simulation is used by 26 states, while 23 states use situational tests. 

3. Thirty-eight states use actual performance tests. These range from local, district- developed 
tests to sophisticated tests such as SOCATS. Most use student performance in the classroom, 
laboratory, or shop as a means of assessment. In some cases, these tests are performed at the 
job site. 

4. Nearly one half the states use a combination of testing processes to determine mastery of 
skills. Examples include: 

a. Ohio uses both cognitive and performance testing to determine competencefor the 
Career Passport, a document which accompanies the student to business or industry. 
Ohio uses ACT Work Keys as an assessment process. 

b. Oklahoma is using cognitive scenarios and performance to test competencies. 

c. Utah uses simulation and situational tests. 

d. Kentucky is presently developing performance tests. 
Vocational-technical education during the past 10 years has developed curriculum in applied 

physics, math, chemistry, biology, and communications. States pooled resources and, through 
consortiums with the Center on Occupational Research and Development (CORD) and the Agency for 
Instructional Technology !AIT), developed these materials. 

Assessment of these skills, along with occupational skills and employability skills, is essential. As 
seen in Table 17, states include in their assessments of vocational-technical students, academic 
competency lists in math, science, communications, and reading. Forty-two states test communications 
and math skills, 40 states test science skills, and 35 states test reading skills. Examples of states which 
test for ail areas of academics include the following: 

1 . Louisiana inc'udes all of these areas in the curriculum, as well as testing. 

2. Maine has developed the Maine Education Assessment (MEA) to incorporate these skills tests. 

3. Illinois is establishing a testing program to incorporate ail areas. 

4. Pennsylvania tests for all academics, as well as occupational competencies. 

5. Minnesota assesses academic and occupational skills, but tests are designed in each local 
district. 

Employment-related skills are assessed in several areas, including problem solving, thinking skills, 
and team skills; resume and application writing; and human relations on the job. 

Forty-four of the states include these areas in assessment of occupational skills attainment 



(Table 17). 




ERIC 



113 

Testing and assessment of skills is tied directly with instruction in most states. There is very little 
evidence to indicate that any one state or field of vocational-technical education is moving to large scale 
independent assessment of skills. Certifications of skills practices are usually in the form of profiles, check 
sheets, certificates with lists of competencies attained, computer printouts of skills mastered, or similar 
records. Ohio includes its certificates and lists of competencies attained in the Ohio Career Passport. 
Oklahoma and Arkansas have strong testing programs. Most portfolios and records of competencies 
mastered are completed by the local secondary or postsecondary program. As indicated earlier, some 
states are moving to a valid and reliable testing system, but this is not yet widespread throughout the 
country. The costs for such a system may be prohibitive for small states, making some form of consortia 
the most practical approach while, allowing states to retain some flexibility to meet special needs within 
their states. Local systems are closest to the student but have the least time and resource capability to 
ensure consistency and validity. 

TABLE #17 ACADEMIC COMPETENCIES AND EMPLOYMENT-RELATED SKILLS INCLUDED IN 
OCCUPATIONAL SKILL ASSESSMENTS 



Related Academic Competencies 


Percent of States 


Math 


81 


Communications 


81 


Science 


77 


Reading 


67 


Other 


8 


Employment-Related Skills 




Problem Solving and Related Skills 


85 


Resume and Application 


83 


Human Relations on the Job 


83 


Other 


19 



Note: State responses indicated inclusion of multiple categories of related competencies and skills in occupational assessments. 
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Use of the Standards 

The greatest usage of the occupational skill standards is by local districts. They are used equally 
by administrators and instructors. Secondary schools in 42 states, postsecondary institutions in 38 states, 
business and industry partners and apprenticeship programs in 27 states, and Job Training Partnership 
(JTPA) programs in 32 states use the occupational skill standards. The partnerships with business, 
industry, apprenticeship, and JTPA provide an opportunity to share occupational skill standards as a 
common base for skill development and employment practices. (Figure 8). 



FIGURE 8 GROUPS USING OCCUPATIONAL SKILL STANDARDS 



55 
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The Performance Standards that were established under the Perkins II require vocational-technical 
education programs to be evaluated by a giver, set of standards. One of the challenges the states are 
facing is how to mesh the specific occupational skill requirements for individuals with the performance 
standards. There is a need to link the two sets of standards. Clearly, it is too early for all of the details to 
have been settled, but states indicate that a strong reiationsnip betwesn the occupational skill standards 
and the performance standards exists. Thirty-three states require locai program? through tlie Performance 
Standards to use occupational skills and competencies. In addition, 41 states require, as a part of the 
performance standards, vocational-technical education programs to have business and industry 
partnerships. Therefore, a strong relationship is being built by states between performance standards and 



ERLC 



125 



115 

occupational skill standards. 

As can be observed building blocks exist within the education system to promote, cultivate, adjust, 
and expand the development of a national voluntary system skill standards systems. However, as will be 
noted in the review of Volume M of this report that provides more detail information regarding current 
activities in the states there are some barriers that will need to be addressed. Nettlesome issues such as 
how to assure the instructional staff (both academic and vocational) is sufficiently knowledgeable of the 
needs of industry has been identified as a substantial barrier. Development of a more sufficient and 
affordable assessment system that carriers with it the portability deemed of value by industry will need to 
be addressed. Encouraging usage by more local school districts and other education and training providers 
all present examples of issues that need more attention. 
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CHAPTER VIII 



INDUSTRY DRIVEN SKILL STANDARDS SYSTEMS 



A. INTRODUCTION AND BACKGROUND 

it has been shown that industry representatives have been involved in providing education 
institutions with information necessary to develop curriculum to meet the changing needs of the 
workplace, through consortium of states, or within states. There are additional venues within which 
industry associations and organizations representing a specific occupation influence the offerings of 
educational institutions or training programs, or the assessment of an individuals skills and knowledge. We 
now turn our attention to the role of national industry and occupation focused organizations. It is 
important to note at the outset that the term "skill standard" has traditionally been a technical one - not a 
policy term. Policy discussions tend to use the term "credentialing", and what follows discusses the issue 
using that term. 

The certification of competence of individuals is embedded within a larger framework dealing with 
credentialing that touches every professional and academic institution. Credentialing activities can include: 

1. prescribing education and experience qualification for certification candidates; 

2. establishing for potential accredited institutions qualifications for curriculum, faculty, and 
facilities; 

3. administering competitive exams; and, 

4. conducting assessment visits. 

Proponents of the private sector taking the lead to perform many of these functions argue that it is 
more effective for the private sector to perform these self-regulatory functions. (Jacobs: 1992) The 
essence of these arguments is that self-regulation prevents costly and cumbersome government regulation 
and that the privately driven approach to credentialing requires the providers of the service to stay close to 
the changes in the marketplace. 

Self-regulation of credentialing, is very much an American strategy. By far the typical pattern in 
other industrialized countries is for government to perform the credentialing functions. Yet even in America 
it would be erroneous to assume that government is not a substantial player in the field. At times 
government has been the initiator, supporter and indeed partner in this complex arena. 

The industry associations that have become involved in the development of both accreditation and 
credentialing programs have entered into the arena of regulatory policy and therefore have had to pay 
attention to the key roles of participants, the representation of interests and the design and performance of 
the system of rule making, monitoring and enforcement. As Lad notes, ...self-regulation is public policy, 
requiring scrutiny of how decisions are made and how they affect the public. (Lad:1992) 
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The Focus of this Study 

The growth of private organizations engaged in credentialing has expanded considerably since a 
survey was conducted in 1965. At that time 120 private organizations, were providing such services. 
Estimates now are that this number is five to ten times higher. {Jacobs, 1 992) The results of this study will 
not yield a definitive number for a variety of reasons. A key reason is that the focus of the study has been 
on the occupations not normally requiring a four year college degree or more as minimal entrance 
requirements. Therefore, information was not sought from organizations that classify themselves only as 
professional societies (e.g. social workers, lawyers, medical doctors). 

The study centered attention on the national trade associations that, as part of their mission, 
directly provide or sponsor, through an affiliate or subsidiary arrangement, a certification service. These 
associations were selected using the Encyclopedia of Associations. Please be aware that this publication 
allows the organization to self-select how they wish to be described. Because of this policy, some 
organizations fitting the definition of a professional society are included in the listing (even though there is 
a separate category for such bodies by the publishers of the Encyclopedia). 

The journeyman card represents a form of certification that does not fit into the trade/professional 
association category. However, the apprenticeship and upgrading programs jointly sponsored by 
labor/management committees were considered a part of the industry driven skills standards development 
"system" for the purpose of this study. The examples of apprenticeship programs were culled only from 
those programs that have nationally recognized standards (excluding the military which has the largest 
number of nationally recognized programs) from the Bureau of Apprenticeship and Training (BAT) of the 
Dept. of Labor. 

It is recognized that individual companies have established internal skill standards for their own use 
and these are often considered to be proprietary. They have been developed to assist the firm's 
competitive position in the marketplace. The study was not designed to capture any of these firm- 
specific skill standards nor was there a review of the apprenticeship programs/standards that are sponsored 
by individual companies. 

The study design was constrained by a Federal Office of Management and Budget rule that no 
written surveys could be administered without permission and the time lines and resources available for the 
study made the use of standard survey methods not feasible. Given such constraints the willingness of 
many associations to provide off-the-shelf information regarding their certification programs was 
impressive. This information was enhanced with the addition of material from such organizations as the 
American Society of Association Executives, the National Organization for Competency Assurance, the 
National Commission for Certifying Agencies, information in the files of both the Departments of Education 
and Labor and the files of the study team. 

From these various sources a substantial (one could even say overwhelming) amount of 
information was collected. The descriptive material is in VOLUME III of this report. Below is a synthesis of 
the findings. There emerges a portrait of how the private sector is deeply engaged, or in some sectors not 
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engaged, in the development and maintenance of one or more forms of credentialing. 

By far the two most predominant forms are participation in the program accreditation process and 
in the certification of workers, normally through the use of nationally validated tests and other forms of 
performance assessment. 



B. COMMON PATTERNS 



Gaps and Coverage 

There are major industry sectors that do not have a tradition of promoting industry wide skill 
standards. Among these are agriculture, mining, retailing and large portions of manufacturing. Clearly 
these industrial sectors employ certified individuals such as engineering technicians, records management 
specialists, professional secretaries and a host of others but these represent occupations that cross 
industry boundaries. Manufacturing firms are often involved in sponsoring apprenticeship training programs 
that are frequently firm-specific and enrollees are culled from the existing workforce. Most enrollees have 
gained some level of seniority and the course of study and the ultimate certification may or may not reflect 
cross enterprise standards. 

It is possible that the agriculture sector has not felt the need to become deeply involved in either 
accreditation or certification due to the traditionally strong base of education's (institutional) support for 
training employees for this sector at both the secondary and post-secondary levels. The retail sector may 
not have perceived a need to become deeply engaged due to the way the industry has evolved with a 
strong reliance on hiring a large cadre of individuals in the "youth labor market" as well as a large number 
of part-time workers. 

By far the largest number of certification programs are directly related to occupations and 
industries where there has been (a) intervention by government to regulate the industry or (b) threat of 
government to regulate the industry and certification of individuals has been the route selected by the 
industry to impose some form of self-regulation upon itself. Health care and real estate reflect examples of 
the first type and direct selling is an example of the latter model. 

Another common pattern is that most of the certification programs reflect specialties or market 
niches. Appraisers and financial and insurance service certification programs exemplify this pattern as do 
certifications for apartment and other types of commercial building managers. 

Many of the certification programs are geared towards occupations where there is not a large 
concentration of workers performing the same function at a particular work site. Again, this reinforces the 
specialty pattern. In some instances these specialty workers must be familiar with laws and regulations 
such as building codes. 

Table 2 shows the distribution of existing skills certification programs by major industry and 
occupational group. The numbers in each cell identify the number of separate industry or occupational 
association based certification programs identified by the IEL survey. The shaded cells indicate 
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industry/occupation pairs containing few workers in the labor force: The indicated industry sector 
accounts for less than 5% of total employment in the occupation group. A total of 168 certification 
programs were identified. Over half (85) are primarily associated with occupations in the Service Sector of 
the economy. The second largest number (25) were found in the Finance, Insurance and Real Estate 
Sector. Twenty programs were deemed to be cross-cutting with respect to industries: no single industry 
could be identified as the primary focus of the program. These cross-cutting programs were predominately 
associated with management specialties and administrative support occupations. Management-related 
skills certification programs comprised the largest single occupational group (56), followed closely by 
professional specialty occupation group programs (31). It appeared that many of these programs served 
college educated workers. The industry and occupation categories used in the table correspond to 
Occupation Employee Survey aggregate groups definitions. 

The Role of Industry Associations 

In very few instances was there found a single industry association that represented the total 
industry, offering the only certification programs used by all of that industry. Often the inclusive industry 
certification programs are in areas where total employment is relatively small, such as the association 
representing the locksmiths or farriers (horseshoers). However, unless the occupation is tied to licensure 
requirements (discussed later) full certification coverage of the practicing employees does not occur. 

There are other examples of certifying organizations that come close to representing the total 
occupational grouping (not the industry per se) such as the American Welding Society that provides 
support to a wide range of sub-industry groups that require competencies in welding. 

These findings to coverage are not surprising given that most of the industry associations are, 
themselves, specialty organizations. While they may have special membership categories or act as the 
convener of several related organizations, the particular specialty that is the association's focus will drive 
certification programs. 

There is a strong pattern of influence that emanates from a primary profession such as 
engineering, accountancy, or medical over the certification programs that recognize the technical workers 
within the same field. For example, many of the allied health fields are clearly modeled after the medical or 
dental preparation programs for doctors where there is a firm belief that the core of the profession is rooted 
in a common body of knowledge that must be acquired as a pre-condition for being eligible for an 
examination or some other form of certification. In such instances there is often a strong relationship 
between an accreditation body and the certifying body even though they are separate entities. 
Additionally, these certification programs are often directly linked to a requirement that an individual must 
pass some form of licensure examination (i;. order to practice in a given state or provide services for a 
federally funded program such as public housing management or approving loans for a federally insured 
program). 
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Other certification programs appear more driven by the technologies of the workplace and while 
the association is often involved in the promotion of education programs (their own and others they 
recognize, including public education institutions,) they have not developed formal accreditation programs. 

Characteristics of the Certification Programs 

In many ways the current state of affairs means that the old adage "let the buyer beware" holds 
sway when discussing certification and credentialing programs. This observation is not made to pass 
judgement on any organization or institution but simply to note that one cannot assume that a person 
claiming to be certified or credentialed has completed an assessment of skills and knowledge in a program 
that has validated the standards or the assessment instruments. Nor is it always the case that a common 
body of knowledge has been developed that is nationally recognized as reflecting the essential 
characteristics of an clusters of common occupations. 

Even with these notes of caution there is much to be said for many of the programs that have 
emerged overtime. 

The Patterns of Practice 

There are some common patterns in the certification systems. 

1 . Most programs only offer one or two recognitions; 

2. Most are linked to time in the workplace; 

3. Most have alternate paths that allow a candidate to credit schooling (particularly from an 
approved program of study) with less time spent in the workplace prior to being considered 
minimally qualified to sit for an exam or other form of assessment; 

4. The overwhelming model is to assess knowledge through passing a paper and pencil test. 
These can take anywhere from an hour, to half a day, to two full days per test; 

5. Most have some form of required recertification based on continuing education and professional 
development activities; these ordinarily must be achieved within a specified time period. However, 
there is a growing pattern to require testing or some form of documented performance assessment 
for continual certification; 

6. Most have permitted a time limited period for "grandfathering" members of the profession 
without requiring the individual to sit for the exams. 

7. Most have developed a core body of knowledge that is required in order to qualify minimally for 
being considered a candidate (all of the nationally significant programs have done this). 

8. Some have established linkages with recognized organizations such as the American 
Council on Education in order to have the passage of an exam count for college credit. 

There are some that have established career paths and varying levels of recognition beginning from 
entry-level competencies to master levels, as will be noted in some of the examples discussed in Volume 
Ml. 
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As the programs grow in number and mature, the issue of recertification is gaining increased 
attention. This is particularly true in occupations where technologies are changing rather rapidly. The 
concerns center around maintaining credibility ana public confidence in the certification. The most typical 
form of keeping up has been the requirement for certificants to take continuing education courses. 
Individuals are encouraged to take courses offered by the sponsoring association. However, it is a dubious 
legal practice only to recognize courses from a single provider so many organizations have now 
"recognized" other education providers' programs or have said the courses can be acquired through a 
formal accredited program that is accepted by the industry and COPA. 

Management and Financial Issues 

While the role of the sponsoring industry association varies considerably, many of them derive 
substantial income from the sponsorship of education related activities that sometimes tie into the 
credentiaiing program. However, as was discussed in Chapter VI it is legally prudent to maintain 
administratively separate and independent governance organizations that have oversight responsibilities for 
the certification and accreditation programs. Clearly not all of the associations have been as careful about 
such interrelationships as they should be in order to avoid the threat of legal review by either the Federal 
Trade Commission or the Department of Justice. However, many of the certification programs that could 
be called "nationally significant" have generally complied with these self-regulatory standards. 

It is often the case that non-profit organizations are established by a single trade association or, in 
some cases, by several trade associations to support research, development and certification program cost, 
in some instances these non-profit organizations have the responsibility to set product standards as well as 
establish skill standards for individual certifications. 

There is a specialized industry of test developers and test givers that are often called upon to 
provide the technical expertise to conduct the job analysis, develop the test items and often adt. .mister the 
testing process and sometimes maintain the registry. The association promotes the certification program, 
provides the experts for reviewing test items and assessing essays or other forms of proof of competence 
such as a business plan, etc. 

When a choice is made to contract out the development of the tests and assessment instruments, 
a wide array of private and or non-profit organizations exist to provide support. Examples of these include, 
The National Occupational Competency Testing Institute (NOCTI), Educational Testing Service (ETS), 
American College Testing (ACT), American Institute for Research (AIR), the National Assessment Institute, 
and the Professional Examination Service. 

A "new to this country" organization is the North American subsidiary of the Cities and Guilds. 
This organization has its roots in the United Kingdom (U.K.). One of the oldest independent individual 
certification organizations in the world, it's approach in this country will follow that of the U.K. and it will 
offer its own certificates in a wide array of occupations. 



As noted earlier, recertification is becoming a larger issue for many of the programs, yet many 
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associations have found that members are often resistent to the requirement that some form of testing be 
a part of the continued recognition. Without the passage of some test and other assessment of skills it is 
necessary for the sponsoring certifying agency to keep records of proof that the individual has complied 
with the recertification requirements. This task apparently has proven to be somewhat challenging for 
some of the smaller organizations and they have adopted a process that notifies the registrants that a 
certain percentage of those certified will be audited each year and they must show proof they have lived 
up to the requirements. 

There is a reality for any voluntary certification program that promotion is a part of the 
management tasks. Individuals must be encouraged to become participants and employers must be 
encouraged to utilize the certification in their hiring and promotion policies. Trade association meetings are 
often used for both types of promotions by certifying bodies. Due to this dependence there is sometimes 
tension between the certifying organization and the sponsoring trade association that would like to have 
more influence over the certification program or share in the fiscal rewards of a successful certification 
program. 

Perhaps one of the most impressive features of certification programs is the willingness of 
volunteers to contribute substantial time and energy to support the effort. It would be impossible to 
estimate the value of these contributions. Without the cast of thousands that often give of their own time, 
it is probably fair to say that there would be no self-regulatory/voluntary certification programs in the 
country. However, volunteers cannot do it all. Most programs are eventually able to become self- 
sustaining by charging a variety of fees for different parts of the certification program. 

A review of the fee structures show several common patterns. It is typical to see application fees, 
study guides and textbook fees, individual test fees, and continual registration fees all used to help defray 
costs. There is also a pahern of higher fees for programs that do not have a large potential pool of 
applicants or where the Opportunity for suostantial income is possible. Or a higher fee will be charged to 
the applicants in order tp recover development cost. This is perhaps not a substantial problem for higher 
income occupations but clearly it is of concern for occupations where the wage rates are moderate or low 
for entry-level workers. 

In all cases where a trade association is a key sponsor there is a different fee pattern for the 
association members versus non-members. These differences must be reasonable or the association can 
run afoul of the federal regulators. 

Speed in the development of programs is directly related to the issue of money. If there is no up 
front "seed money" from either the parent trade association or professional society it can take several 
years for the examination and other forms of assessment to become realities. Staff responsible for the 
development of the programs are often forced to justify developmental costs to the point that sometimes 
corners are cut or programs are abandoned. There are some instances where test development and test 
administrator enterprises have agreed to share the development cost based on the business assumption 
that a sufficient market exists of potential applicants to recover the costs. 
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The total cost varies, in part based on the decision to contract with test developers, but several 
hundred thousands of dollars are required (including voluntary time of experts) to launch a substantial and 
credible program. 

Program Recognition or Accreditation 

One of the stated goals of the federal government's initiative is to promote a competency based 
system that builds, into the education and training curricula, requirements for workforce skill and 
knowledge identified by industry. It has already been noted that the term "competency-based" has yet to 
achieve common meaning among the different stakeholders that must necessarily be involved in the 
development of this new system. This goal, if realized, will represent a sharp break with current practices 
on several fronts. 

The typical pattern that has been promoted by industry when supporting accreditation programs 
has been to develop agreement between educators and themselves on what is often called the common 
body of knowledge required to practice in a profession. This provides the core rationale for the need to 
have specialized accreditation bodies at the post-secondary level of education. In addition to assuring that 
an education institution provides courses that contain the common body of knowledge, accrediting bodies 
also consider a host of other issues. One of the key items they consider is the qualifications of the faculty. 
The accreditors want to assure that these individuals remain knowledgeable of trends in the industry. 
Equipment used in the education process is another item of concern for some accrediting bodies. It will 
often be the case that some form of work site practicum or internship is required of the students in a 
specialty-accredited program. 

At the core of the accreditation process is a self-assessment process that an institution undertakes 
to determine if it is in compliance with the commonly agreed-upon program standards of the accrediting 
agency. After the self-assessment phase, the institution undergoes a peer review conducted by individuals 
selected by the sponsoring body. There are time limits on the approval, normally lasting five years before 
the cycle is repeated. 

This accreditation model contains a substantial set of common principles used by many of the 
nationally-recognized apprenticeship programs. These apprenticeship programs also have developed a 
common body of knowledge; concern with type of equipment used; and, the qualifications of instructors 
have proven to be of paramount concern in many of these programs. Indeed many of the apprenticeship 
program staff often note the most important quality assurance factor in their programs is the instructors. 

These two forms of traditional quality assurance also have in common a program that is essentially 
time based. Additionally, the specialty accreditation and apprenticeship programs are primarily geared to a 
post high school period of training. 

There is a small, and perhaps instructive, emerging effort on the part of some industry associations 
that have traditionally drawn entry-level workers from the ranks of high school graduates to emulate the 
accreditation approach at the secondary level of schooling. These efforts are sometimes described as 
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program certification. They often use the same categories such as instruction, qualification of staff, type 
of equipment, etc. that accreditation programs use to certify a high school program. The curriculum is 
often described as "competency-based" due to its development having invoivsd industry representatives. 
There are however, limited examples of national organization's sponsorship of such efforts. 

For many occupations graduation from an accredited/certified specialty program has proven to be a 
sufficient entry level credential. However, it is instructive to note that, a substantial number of the 
industry trade associations that are deeply involved in the sponsorship of accreditation programs are also 
involved in a certification of an individual's competencies in a separate program effort. As one individual 
interviewed stated succinctly -- an accredited program only allows an individual to learn the certification 
testing process finds out what they learned. (This may or may not be true but it is said nicely and 
summarizes the rationale for the coupling of the efforts.) 

Suffice it to say that one of the more important challenges that will need to be addressed in the 
United States will be deciding what is meant by a "time-based" v.s. what is "competency-based" 
education. 

C. THE APPRENTICESHIP APPROACH 

The cost for the apprenticeship model of credentialing is often supported by a self-imposed tax 
that employers pay as result of collective Dargaining agreements or by the choice to sponsor such a 
program, in these programs, the costs often include the development of curriculum, training of instructors, 
the development of tests. In most of the joint union/employer sponsored programs, apprentices do not 
bear any of the related training costs nor are testing fees required. It is not typical for apprenticeship 
programs to have a set of tests or other forms of performance assessment that come at the end of the 
apprenticeship period. If there is a nationally recognized program that includes passage of tests, they are 
given throughout the apprenticeship period and continued participation in the training is contingent upon 
the passage. However, there are no requirements that an approved program include such a quality 
assurance mechanism and this has, in some instances, caused problems in the "portability of the 
credential" from one community or state to another. 

The certification of competencies that come from an apprenticeship type training program follow 
the first form of credentialing - prescription of education and experience - with the testing closely akin to 
the traditional approach used by education institutions - taking the test at the end a particular course. 
The test may or may not have been nationally validated. 

The Seafarers apprenticeship/upgrading programs are unlike some others, in that in several 
instances, the skills certification are ultimately documented by the government agency, most often the U.S. 
Coast Guard, that directly administers tests to enrollees. 




ERIC 



126 

D. CONCERN ABOUT THE INSTRUCTORS « IN THE SCHOOLS AND AT THE WORK SITE 

There is a consistent theme, indeed concern, that permeates almost all of the industry associations 
that are involved with any form of credentialing - the qualifications of instructors. 

As already noted, several of the national apprenticeship, programs invest considerable time and 
energy in preparing experienced industry workers to perform the all important task of providing the related 
instruction. 

Many of the industry certification programs have developed a screening mechanisms to vouch for 
the quality of instructors that can be used in their own education programs. The specialized accreditation 
and program certification programs also place a great deal of emphasis on the quality of instructors and, in 
some cases, the educational institutions' tenure policies are sharply criticized by members of the industry 
who are not convinced that all of the instructors have "kept up" with the changes in the occupation and or 
industry. 

There is also a related and important issue regarding the quality of instruction that takes place at 
the work site. This concern has beon addressed in varying ways. For programs that have prescribed 
education and experience qualifications for certification candidates - internship, cooperative education, 
apprenticeship program - this has often meant that the individual who is the work site supervisor must 
themselves, be "certified" in order for the student to gain either the needed credits for graduation or 
progression. 

As will be noted in several of the examples of industry supported programs discussed in Volume II, 
the search for the necessary training support at the work site continues to be a problematic issue. 

E. RELATIONSHIP TO LICENSING 

The merits of licensure are not the focus of this study but it is impossible to ignore the connections 
of voluntary credentialing programs with that of the government authority to control who is allowed to 
practice in an occupation. 

Complex is the operative word regarding the relationship of licensing and voluntary credentialing 
systems. The world of licensuring is complex by itself. The construction and building maintenance 
industries are often required to meet local licensure requirements; however, most of the licensure standards 
are established by state governments and permeate several industries such as health, allied health, 
accountancy, real estate, some financial services, cosmetology, and the legal professions. 

The federal government also is involved in some licensure requirements in such areas as nuclear 
power, marine occupations, air traffic control, and others where this level of government rule making 
supercedes state authority to protect the health and safety of public. 

Often associations representing a particular occupation aggressively seek the government-required 
licensure by way of controlling entry into the practice of the occupation or to help members address 
potential health or safety liability legal claims. 
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There are obviously mixed opinions regarding the role of government. While there are strong 
advocates for minimal government activity in this recognized self-regulatory arena many professional 
associations are strong advocates of licensure requirements and have lobbied state legislative bodies to 
advance this position. There are often contentious debates when states consider such legislation with 
businesses and other closely related occupational interest groups often opposing the sought after 
recognition. 

Many voluntary certification and accreditation organizations have close working relationships with 
state licensure boards. For example, the American institute of Certified Public Accountants (AICPA) works 
In concert with the National Association of States Boards of Accountancy to advise state regulators 
regarding which education programs meet the academic requirements AICPA believes meet the appropriate 
course of study needed by the applicant to sit for the CPA exam. Additionally, AICPA develops and 
administers the nationally recognized CPA exam that is used by the states as proof of competency. AICPA 
is clearly not alone in such a working relationship with the state licensure community. This pattern is 
followed by the health and allied health fields and others. 

In some of the licensed occupations there is a level of disdain on the part of national organizations 
regarding the value and worth of the some of the state exams. The essence of the criticism is that the 
exams often represent the minimal knowledge required to practice the craft or occupation and do not 
address the full range of knowledge required to practice as a full, well-rounded professional. 

These intertwined relationships clearly blur the purist arguments that certification and accreditation 
are totally a self-regulatory matter or that, in some cases, whether certification is truly voluntary - if the 
individual wants to be employed in the effected occupation. 

Table 18 shows the distribution of occupational licensure programs by major industry and 
occupation category. The table was compiled from information listed in Volume III, Appendix B. Each 
licensure program was identified in terms of the industry and occupation groups in which the licensed 
workers were primarily found. The identified licensure programs were analyzed to match primary industry 
and occupation categories. The focus of licensure program selection was to identify programs where 
licensure typically incorporates some degree of skill, competence or knowledge requirement. It was found 
that the largest number were associated with the service industry sector. Professional specialty 
occupations accounted for the largest number of licensure programs. 

F. SOME UNSETTLED AND CONTINUING ISSUES 

Each of the previous chapters as well as this one have noted several sets of issues that will need 
to be addressed as the next steps are taken in the development of a national skills standards systems. 
However, it is important to note a few more that have not yet been discussed. 
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There is no common framework that guides the development and implementation of the programs 
and systems. Common language is a powerful tool but no common language exists between and among 
the programs and the education community at large, let alone among the general public. 

There is also a clear pattern of parsimony. There are few certification or credentialing programs 
that are targeted to the entry-level workforce. Even within industries that hire young first-time job seekers, 
the certification programs do not focus on entry requirements. Also most of the programs do not promote 
a graduation of knowledge and skills to reach the high end or, master level. 

Another substantial concern is competition among professional associations, certifying/accrediting 
groups, and even licensure bodies. Without denying the value of competition and the value of market 
forces there may be a need to set some parameters. 

An illustrative -- though by no means isolated example - of an occupation where accreditation, 
educational requirements, certification and licensure issues are unsettled is the nursing profession. The 
system is driven by the members of the occupation. In 1984 there were 85,000 certified nurses. By 
1987 there were 148,000 and the number is steadily increasing. The federal government supported the 
importance of certification when it required all institutions that receive Medicaid and Medicare funds 
provide assurances that support personnel are certified. 

Twenty-one organizations offer nursing certification. The proliferation of credentialing has led to 
some duplication. A pediatric nurse practitioner, for example, can be certified by two different 
associations. In a few specialties, a master's degree is needed to seek certification. In some, a BSN is 
needed. In most, however, all that is needed to qualify for certification is a RN license and one to five 
years of relevant experience. Certification is normally valid for 2-6 years. In addition to some duplication, 
there is a continuing struggle within this care-giving industry between job categories of providers that have, 
in some cases, spilled over into the certification and licensure arena. 

Sorting out which level of skill and expertise is needed to perform which jobs is often subject to 
debate within the affected workforce and this has spilled over into the licensure and certification arena. 
Within the nursing profession an example of such tension is the relationship between registered nurses and 
licensed practical nurses. The National Federation of LPNs (NFLPN) was founded by LPNs who wanted 
their own voice and their own leadership. The LPN is supervised by a doctor or an RN and cannot initiate 
intravenous drugs or administer blood or blood products. Training is received through a one-year program, 
although the credit hours required vary from school to school. 

In the early 1980s, minimum competency standards included specific restrictions on the situations 
in which LPNs could practice without supervision. At that time, they also began finding it difficult to 
compete with RNs for a limited number of hospital nursing jobs. By 1985, officials of the ANA were 
seriously considering recommending "phasing out" the LPN. The thinking was that nurses needed more 
education to competently care for the new breed of patient, one arriving sicker and leaving sooner. Now, 
with the nursing shortage, the tide is changing in favor of the LPN. 
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Even if there is to be a substantial expansion of industry-driven accreditation, certification and 
assessment systems to improve the "market responsiveness" of the education and training systems, there 
will no doubt continue to be a set of trade-offs that must be dealt with regarding minimum reouirements for 
certification eligibility. Again the nursing profession highlights the dilemma. 

The requirement of the baccalaureate degree in nursing for generalists certification is a policy 
question which has undergone a somewhat extensive analysis over time within ANA. The Cabinet on 
Nursing Practice in its review of the question in 1985 moved to establish a policy, with consensus from the 
certification boards and councils, into place a moratorium on establishing the baccalaureate in nursing as a 
requirement for generalists certification until such times as 75 percent of the RN licensed population held a 
baccalaureate degree in nursing. Exceptions would be permitted when the nature of the practice area is 
acknowledged by broad consensus to require the minimum of baccalaureate preparation for practice in the 
specialty area. 

ANA has generally held to the belief that the generalists areas of certification should focus 
primarily on the more broadly defined areas of practice, where there are identified a sizable number of 
nurses practicing within the area, and where there is a body of knowledge for the practice which can be 
clearly defined as sufficiently discrete from other areas of practice. 

There is some push to require certification for employment in such areas as critical care, 
emergency care, and intravenous therapy. For the most part, employers do not require certification. Given 
the nursing shortage, it is unlikely that employers will require certification in the foreseeable future. The 
credential may help, however, in earning more money and also help an individual be promoted. Some 
hospitals, 13 percent of hospitals with ANA contracts, boost the pay of certified nurses. In Veterans 
Administration hospitals, certification can lead to an advancement of one to five salary levels. These are 
exceptions rather than the rule, however. (Wills: 1992) 

The reliance upon the requirement that individuals have a core body of knowledge prior to being 
eligible to sit for a certification or licensure exam appears to be a rational standard for many occupations. 
However, it appears that in some cases this requirement has been used as a tool to drive individuals back 
into the classroom - sometimes to start all over again even though they have been practicing within the 
field. This apparently is due to the unwillingness of the specialized accreditation bodies to recognize 
course work in even a related field as well as refusing to recognize learning acquired through experience 
gained on the job. This study was not designed to address the merits or demerits of that approach, 
however such practices raise a number of questions about how to create more rational articulation 
strategies within major occupational areas. 

Correspondence from Kenneth Edwards, Director, Technical Services IBEW, provides another part 
of the challenge of the skills standards initiative. "IBEW helps train for 750 occupational classifications 
that are covered by 14,000 to 15,000 collective bargaining agreements in seven or eight major industries. 
Some of these industries are under federal control, such as the Department of Energy, Federal 
Communications Commission, Department of Energy, Public Health Service, Nuclear Regulatory 
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Commission, Food and Drug Administration and the Mine Electric Safety Administration. Others carry state 
electrical licenses. 

Approximately 200,000 of these members are employed in the electrical construction industry 
representing about 40% of the electrical labor force in the U.S. They have anywhere from 1200 to 1600 
certified teachers and many members certified by various associations. 

This story will be repeated again by others as the progress is made towards reaching a common 
understanding consensus regarding the development of a national, voluntary skills standards system. 
Much remains to be accomplished. 
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CHAPTER IX 



RECOMMENDATIONS AND CONCLUSIONS 



A. INTRODUCTION 

All of the research leads to a question: What does this imply for the further development of a 
voluntary skill standards system in the United States. The U.S. Department of Education asked the 
research group to explicitly answer six questions to help understand the scope of the roles for the federal 
government and the private sector in the development and implementation of skill standards to improve 
U.S. competitiveness in the global market. The six questions are: 

1 . What roles should industry and the federal government play in developing a nationwide system 
of skill standards? 

2. What arrangements must the federal government adopt to encourage industry to become 
involved in the development and use of skill standards? 

3. What are the best forms of assessment/testing to use in the skill standards systems? 

4. How could a nationwide system of skill standards affect education programs at state and local 
levels concerning educational program improvement, curriculum modernization, and educational 
choice? 

5. What testing guidelines should be recognized to insure against discrimination based on gender, 
ethnicity or disability, status? 

6. To what extent do the current apprenticeship and cooperative education systems in the U.S. 
provide a framework for setting skill standards? 

Responses to the six questions and the recommendations which follow were derived from a 
synthesis of the experiences in the U.S., six other countries and a study of the Center for Policy Research 
of the National Governor's Association (NGA). The NGA project was a case study of 19 states that have 
undertaken state based skill standards programs. 

This study was primarily a descriptive study. The study's focus, by contract design, was to 
provide information to the federal government about what exists. It was not a search to find the best 
practices based upon a set of evaluation questions, with the exception of the literature review of the 
current practices in job analysis and assessment. That review provided some suggestive guidance about 
best practices, but a wide range of unsettled issues remain in both of these areas. 

It is important to understand the suggestions are rooted in assumptions about an "ideal" client- 
centered skill standards system where individuals and employers are considered the client. Our 
assumptions about an ideal system also reflect the key lessons learned from studying the experiences of 
other countries. The five key lessons from international experiences are: 
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► There must be a common framework that includes common language for the development of 
skii! standards; 

► Government must invest, at least developmental dollars, in assisting the private sector establish 
a comprehensive set of services; 

► There must be true consultation processes that involves ail of the stakeholders. Clarity and 
agreement will need to. evolve regarding the roles of different levels of government, the types of 
support that will be needed for the education and training institutions, and the new forms of 
support that will be needed to assist learning in the workplace. This cannot be a top-down system; 

► Substantial support must be provided to the development of the public sector, education, and 
training providers; and, 

► The federal government's most critical roles are that of an enabler and of a consensus builder. 
The Ideal System 

The client centered criteria that have emerged from the combined sources has been condensed to the 
following criteria for the ideal client-centered skill standards system. Our specific assumptions are that the 
system would: 

1 . Not be bound by individual student/workers age and would be widely accessible to young 
people and adults; 

2. Not be bound by type of institutions providing education or training and flexible in response to 
changing, and sometimes different, needs of individuals and localities through a variety of forms 
(e.g., education, full-time and part-time training); x 

3. Would be able to meet the needs of individuals regardless of type of education and training 
they are pursuing (e.g. initial preparation, continual, upgrading, or remedial); 

4. Would allow career paths within and between industries; 

5. Be explicit so that firms, education, training providers, and individuals know what the standards 
are and where information about them can be obtained; 

6. Be a competency-based system; 

7. Have a formal assessment and certification of an individual's skills documented by a third party; 

8. Be a progressive system so that people can build upon blocks of competencies and adapt to 
technological and market changes, to improve their prospects or to explore their potential; and, 

9. Have a common framework and use common language when describing skill levels across 
industries and occupations in order for both individuals and employers to easily understand the 
expectations and value of the use of the standards. The framework should progress from initial 
(entry) qualifications through several levels to mastery and/or specialization recognition. 
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QUESTION 1: WHAT ROLES SHOULD INDUSTRY AND THE FEDERAL GOVERNMENT PLAY IN 
DEVELOPING A NATIONWIDE SYSTEM OF SKILL STANDARDS? 

Industry must be in the recognized lead role for the development of the standards and the federal 
government must help facilitate such development. It is essential to the success of any skill standards 
system that the employer community recognizes the value of these measures to their economic success 
and that they recognize and accept them as theirs. 

The federal government has the critical function of being the enable.* and promoter of consensus. 
This function cannot be overemphasized. Each of the other countries learned how essential it is to have 
quasi-independent organization(s) that are charged with the responsibility to bring together the various 
stakeholders. This has proven to be a substantial, but achievable, challenge in the other countries that are 
all smaller and less complex than our country. Our study shows that structures operating in the United 
Kingdom and Australia to be the most suitable for emulation in this country. In both instances the 
organizations created have a limited charter with a limited budget. 

As of this writing a Clinton Administration proposed piece of legislation is being actively considered 
by the U.S. Congress to establish a National Skill Standards Board. The comments do not address the 
details of the legislation, instead focusing on suggested actions the proposed board, as well as, other 
federal government stakeholders should address to implement a nationwide skill standard system. 
Some of the essential "first order" tasks, we believe, should be: 

Establish a framework for generating valid and reliable skill standards, assessment and certification 
systems. 

The National Board should develop the framework and requirements for nationally recognized skill 
standards. A common framework is essential in order to facilitate the standard setting process through the 
use of standardized language and format. The framework must consider what criteria will be accepted for 
recognition of valid and reliable standards. Review and updating schedules should be created. Standards 
should, when possible, be benchmarked to international standards such as those promulgated by the 
International Standards Organization. Criteria for recertification of an individual must be addressed. 

The current U.S. Department of Labor and Education grantees form a natural core of organizations 
to help assist the Board in the development of this framework. Organizations that have successful 
experience in developing and maintaining third party standard setting and assessment programs should be 
consulted. 
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Establish common levels of qualifications. 

The U.S. will need a progressively complex set of levels of knowledge and skill mastery required 
for individual career entry and progression into the highest levels of his/her chosen career path. Particular 
attention should be given to establishing criteria for recognition of occupations that cross industries. 
Within the broad-based levels, units and elements should be identified so that individuals will have the 
opportunity to be assessed for competencies within a single level and accumulate credit over time and in 
different settings. 

Establish procedures for benchmarking to international standards and for upgrading and maintaining the 
skill standards continuously. 

Skill standards will only lead to higher skills and assist in the improvement of the economic position 
of firms and individuals if they are benchmarked against the highest international standards. Skill 
standards must never be assumed to be static 

fixtures. There should be a planned process for continual upgrading in response to technological change. 

Establish criteria and recognition procedures for organizations that develop standards. 

No one current industry or occupational classification system is fully satisfactory for describing 
either an industry or an occupation. However, the Board should establish a set of criteria that it can use to 
examine an industry group that voluntarily comes together to promote standards within an industry. The 
criteria should include assurances that any such organization recognized represents all aspects of the 
industry, that geographic representation exists and membership includes incumbent workers. 

Particular attention should be given to establishing criteria for the recognition of broad based 
occupations that cross sectors. Pooled representation from several industry groupings may be the most 
feasible approach to jointly develop common standards, where appropriate. Obviously there are several 
occupations, such as secretarial services, that permeate all industrial sectors. These broad, cross-cutting 
occupations, need to be identified and then representations from all of the industries involved in agreeing 
upon the general skills as well as the specialist skills (e.g. legal, medical, financial services) could, if 
necessary develop their own specialized standards. 

Once recognized or "chartered" to develop standards the industry group should develop standards 
that meet validity and reliability criteria. The skill standards should be submitted to the Board for the 
purposes of being "accredited" as standards recognized by the Board. 

When developing the criteria the Board will need to address how to assure there will be a sense of 
ownership, on the part of employers, in the organization(s) that are selected for the standard setting 
process. Without a sense of ownership on the part of employers, within each industry, it will be difficult to 
get employers to share in the cost of the development and maintenance of the skill standard setting and 
certification efforts. 
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Establish criteria and recognition procedures for organizations which design and award the certificates and 
insure the quality control over the assessments system. 

This recommendation is predicated on two assumptions: (a) a third party assessment process is a 
component of the skill standards effort, (b) several different organizations can and will be involved in such 
assessments. However, there should be agreement on the best ways to describe and assess performance 
levels for the complex skills required in the modern workplace. 

The organizations which establish the standards could be in the position to conduct the third party 
assessments or, as is often the case in several trade association-sponsored certification programs, another 
organization conducts the assessment for the program. Several different permutations are possible, but we 
believe the National Board will need to address criteria for the recognition of third party assessment 



A system of easily accessible, cost effective assessment centers that are staffed with trained 
proctors should be created throughout the country. A variety of trade and professional associations have 
substantial experience in this area that could benefit the National Board. It is also possible to use publicly 
funded labor exchange offices and/or education and training organizations facilities and staff as is done in 
Japan. The state's adult education programs in this country have a working agreement with the American 
Council of Education to administer the GED tests that may provide insights in this area. 

Establish procedures that ensure the security of individual records of persons that have been awarded 
recognition of having achieved the qualification(s). Also develop a system that will ensure easy access to 
the standards themselves. 

In order to protect the individual and promote utilization of the standards we have identified these 
issues as two early tasks the proposed National Board should address. 

The report of findings from assessments needs to be generated in a form that is easily recognized 
and is portable from one place to another across the country. 

Any type of certification program must adhere to civil rights guidelines and standards established 
by case law. This means that documentation for all types of assessment must be generated and maintained 
long-term. Early and thoughtful consideration on how to develop and maintain national registries is needed. 

It seems logical that at the same time the maintenance of individual records is being considered 
that attention should be given to ensuring there will be easy and affordable access to the standards 
themselves. The United Kingdom has chosen to vest the functions of managing both of these information 
systems within the charter of the National Board. This may not be the best approach for the U.S. but some 
system should exist. 

Establish procedures for disseminating skill standards to employers, education and training providers, 
assessment systems, and individuals. 



Voluntary skill standards have meaning when they are widely used. Considerable attention must be 



providers. 
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given to promoting use of the standards. Such promotion should be rooted in fact and detail the benefits to 
individuals, employers an-i education and training providers. 

During the eariy stages of the effort developing trust is essential. The National Board should be a 
key player in developing and assuring there is an open and clearly understood process for consultations 
among the stakeholders. Employers, unions, units of government, educators and training providers must 
have formal means by which their concerns are heard. Membership on the Board from the various 
stakeholder communities cannot be the only way to be heard. Procedures adopted by the Board should 
include links to other formal commentary and review processes already in place by the federal government. 

The establishment of a Board is but one part of any skill standards systems. The federal 
government will need to support the effort in multiple ways. Identifying how all the key parts of the 
federal government can promote the use of the skill standards initiative is critical. If history can be used as 
a guide, this will not be easy, even within one department - let alone across several departments. Local 
and state governments as well as industry and unions will be looking for evidence that the federal 
government is ready to act as a leader. 

There will need to be substantial attention given to dissemination linkages with and through state 
governments. One of the primary users of the skill standards will, of course, be education institutions, 
training organizations, career counselors and employment placement staffs. Youth service organizations 
should have easy access to the skill standards. States are the major regulators of occupations (licensing 
and certification) and these regulators also need the information. 

Several options are available to assure wide dissemination of the skill standards to these various 
constituency blocks and through them to individual consumers. Consideration could be given to having a 
Governor designate a dissemination agent which has the capacity to reach into these various communities 
of interest. Dissemination of information in formats useful for different consumers will not be a free 
commodity and some "costing out" should be undertaken. 

Reaching into the employer and union communities will be equally essential. General business 
organizations (with a special emphasis placed on those reaching small business) as well as industry-specific 
associations should all be used as disseminators of information about the value of skill standards. 

Develop a shared development and maintenance funding strategy. 

The proposed Board, in consultation with the "stakeholders", and most importantly federal offices 
and agencies, should develop a shared development and maintenance funding strategy. 

The development of comprehensive and nationally applicable skill standards will require support 
from both the pubic and private sectors. This support can take many forms but without it nationwide skill 
standards will not exist. 

Most of the private sector support for the development and maintenance of skill 
standards/credentialing programs comes from "parent organizations" that are responding to some form of 
economic imperative, or as an act to prevent the impositions of federal regulation over the industry. 
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Individuals are willing to spend money for a third party assessment/credential when they consider it to be 
in their economic self-interest. The proposed voluntary skill standard system is not envisioned as being 
driven by these private motives. 

The envisioned system incorporates additional "public good" imperatives. This will require giving 
attention to sorting out who should pay for what (employers, individuals, and government.) It is not 
presumed that funding from the proposed Board can or should be the only source of government funds 
needed to promulgate the effort nor is it assumed the private sector can or should pick up the total costs. 
For example, federal vocational education and job training funds can be used to help share in the costs. 

QUESTION 2: WHAT ARRANGEMENTS MUST THE FEDERAL GOVERNMENT ADOPT TO ENCOURAGE 
INDUSTRY TO BECOME INVOLVED IN THE DEVELOPMENT AND USE OF SKILL STANDARDS? 

Note: For purposes of this discussion the term industry is meant to include employee organizations 
as well as employers. 

Millions of volunteer hours are donated each year in the development of validated skill standards. 
We see that from both the education and industry-driven skill standards programs across the country. It is 
hard to imagine that it will be difficult to garner industry involvement in the development of the standards. 
Financial support will be needed to encourage the development of comprehensive skill standards that will 
assist the novice as well as the specialist/master. Ensuring a comprehensive system, is we believe, one of 
the key reasons for government action. This is a "big gap" so to speak. 

Many of the private credentialing programs focus "in the middle" of the career path ignoring the 
entry as well as the mastery levels. A common progressive framework and common nomenclature is 
needed to connect career pathways. 

Gaining industry participation in the development of the standards will not be a substantial 
problem. The public role, of helping to establish a common framework and common nomenclature, we 
believe will be accepted by industry because thy they are accustomed to the use of standards. 

Demonstrate the value of skill standards to encourage use by employers. 

It is to the second part of this question the federal government asked us to address that our study 
suggests substantial attention is required. Use within industry may prove to be problematic. There needs to 
be an aggressive "show them and help them" strategy to encourage industry to use the skill standards 
systems. 

Employers will need to be given substantial evidence that the use of skill standards will increase 
productivity within the workplace. There is scattered evidence in the United States that the use of skid 
standards does increase productivity. Evidence drawn from trade and professional associations should be 
compiled and disseminated widely throughout the business community through the Departments of Labor 
and Commerce and by those responsible for promoting the Malcolm Baldrige Quality Award. 
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Develop support materials to focus use of skill standards in the workplace. 

In addition to "show them" strategies a great deal of "help them" work has to be considered with 
a strong focus on the front line supervisors and managers. 

There is substantial evidence that it is costly, complex, and often beyond the means of most 
employers to provide the appropriate focus on workplace-based use of standards. This includes assessing 
real skills and knowledge of workers, monitoring progress, developing and utilizing competency based 
training materials, organizing rotational learning opportunities and more. Strong consideration should be 
given to providing support, once the standards are established, to organizations that represent a significant 
portion of an occupation or industry. This support should include the development of workplace-friendly 
materials. 

Efforts should include development of easy to use manuals that would help assess, in a non- 
threatening fashion, current workers skills based on the standards. Guides that suggest ways to organize 
the workplace to help make maximum use of those employees with documented the skills and training 
materials for work-site or off-site that adhere to standardized language format can also be useful. Training 
materials should be developed with the end user in mind using cost-effective, computer assisted instruction 
whenever possible. 

As third party assessment programs evolve, materials should be developed that affirm work-based 
skill development. 

Provide leadership and support to help employers comply with civil rights laws. 

A major barrier to implementing standards by employers is the fear of civil rights violations. The 
federal government has been hesitant to provide technical support to help resolve such fears. The National 
Board, and other parts of the federal government, should help support the development of technical and 
training materials that will advance the cause of equity, access and quality standards simultaneously. 

It is not possible to estimate the cost of the development of such materials but the price tag will 
not be small. Sharing costs between government and industry based networks should be considered in the 
development of technical assistance materials. 

The federal government has not been a major actor in the development of work-based learning 
materials, but it is possible to use several sources of funds to help supplement the cost. For example, 
funds from Work-based Literacy Programs, Adult Education, Bilingual Education, the Job Training 
Partnership Act, the Carl Perkins Act, the Small Business Assistance programs, and the programs that 
support the Advanced Technology Centers could all be applicable. This recommendation is feasible 
(without violating the discreet purposes of each authorizing legislation) because one of the great 
advantages of common standards is the it allows a wide range of programs and providers to "plug them 
into" their specific mission and purpose. This would allow separate federal initiatives meet their individual 
program goals. 




ERIC 



140 



The federal government could enlist the help of state customized training programs and the 
proposed state Human Resource Councils to promote the use of standards in state funded training 
programs. 

Promote the use of skill standards through incentives. 

Other incentives include the shifting of the treatment of training in the tax code from one of a 
current expense to an investment or promoting the creation of a grant/levy incentive plan based on a 
portion of payroll expenses. Clearly, these recommendations are not based on the finding of the study of 
current efforts in the use of skill standards in the U.S. but again, are drawn from the observations of other 
countries. Almost all have sorted out different avenues to promote the sharing of the cost of investing in 
human resource development between the public and private sectors, with the use of payroll taxes being 
one of the more common approaches. 

QUESTION 3: WHAT ARE THE BEST FORMS OF ASSESSMENT/TESTING TO USE IN SKILL STANDARDS 
SYSTEMS? 

There is no silver-bullet answer to this question. No one form of assessment or testing was 
identified as being superior either in the literature search or in the in-depth review of selected industry- 
based skill driven certification programs. 

It is important to note the criticality of first performing a valid and extensive job analysis for the 
creation of standards. Only through a systematically performed job analysis can one insure that the 
standards and aspects of the certification program accurately represent the needs of the industry and the 
work of the job incumbent. Secondly, provisions of the Civil Rights Act, the National Labor Relations Act, 
and other appropriate law associated with assessment and job analysis must be followed. 

It is clear there is a great deal of interest and experimentation occurring in the development of 
performance assessment instruments. While there was no substantial evidence that paper and pencil tests 
should be totally dismissed there is also no evidence they should be the only instruments used. 

Currently, few certification programs rely solely upon written tests. Many programs found the 
expense plus assuring competency of the^judges necessary to administer performance based assessments 
rendered those assessments unfeasible. Many large scale certification programs have resorted to accepting 
time spent in the workplace and/or recognition of some education as substitutes for specific skills 
performance and as supplements to paper and pencil tests. Some require work samples that are judged by 
a panel of peers. Computerized technologies may permit state-of-the-art simulation testing in some 
occupations (assuming costs are reasonable). 
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Establish consultations with experts to study and improve assessments processes. 

Materials collected from trade and professional associations for this study could be used to identify 
organizations using a range of assessment instruments and criteria to award recognition. Based upon such 
a review, a series of consultations should occur with a wide range of experienced individuals to help inform 
decisions about what are the most feasible and appropriate next steps that would help to establish more 
valid assessment techniques. The Board would also be well served as it develops an assessment research 
agenda if it tapped the work of other countries, particularly the United Kingdom that has invested a 
substantial amount of effort in this area. Representatives of civil rights groups, employers, and unions ail 
need to be involved in the consultations and the development of the research agenda. Such consultations 
could help inform the research agenda of the proposed National Skill Standards Board. 

Bring together federal agencies to develop a linked assessment development strategy. 

Promotion of a voluntary skill standards systems is a part of the response to the National Education 
Goals, specifically, Goal 5. Consultations need to occur with the other offices and agencies within the 
federal government that have responsibility for the development and implementation of the various 
assessment efforts embedded in that goal. Specifically, the work of the National Center on Education 
Statistics, that has been responsible for developing the materials to assess college graduates ability to think 
critically, communicate effectively and solve problems should be studied. The work of NCES clearly shows 
the potential to link the research and development of new assessment methods. 

A particular focus of this joint effort should be to ensure that any federal government initiatives 
promoting school-to-work transition will devote substantial attention to improving independent assessments 
of students based upon the skill standards recognized by the National Skill Standards Board. 

As noted in the "ideal" skill standards system the preference would be to develop mechanism(s) 
that provide for independent third party assessments of an individual. Additionally, the third party 
assessments would be widely recognized and portable across state lines and within the education 
community. Substantial work needs to be done in this area. An aggressive R&D effort should be mounted 
to test out various forms of assessments (e.g. performance, computer assisted simulations, work-based 
assessments) and various approaches to establishing links between education based assessments and 
industry validated assessments. 

One final observation, the government, if it wants to set an example, may need to rethink current 
legislative exclusionary clauses that allow government exemption from the same processes that the private 
sector must follow when creating and using assessment and certification examinations. 
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QUESTION 4: HOW COULD A NATIONWIDE SYSTEM OF SKILL STANDARDS AFFECT EDUCATION 
PROGRAMS AT STATE AND LOCAL LEVELS CONCERNING EDUCATIONAL PROGRAM IMPROVEMENT, 
CURRICULUM MODERNIZATION, AND EDUCATIONAL CHOICE? 

If care is taken to involve ail of the stakeholders and if the necessary support systems are 
developed, the impact on the nationwide system will be profound. 

The "Ideal System" is predicated upon an assumption that all education and training providers 
would use competency based curriculum and instructional materials derived from the skill standards. This, 
we believe, supports the concept of choice in the most pragmatic and useful way possible. 

There will need to be a clearly developed consensus regarding the appropriate roles and 
responsibilities of education and training providers in the development and implementation of a skill 
standards system. 

This study, we believe, highlights the fact that the most important functions to be completed by 
the education and training institutions are the delivery of education and training services, not the 
development of the standards themselves. Without strengthening the capacity of the education and 
training system the skill standards "system" will flounder. 

Develop an array of capacity building networks to assist education and training providers. 

Another "first order" of work for the proposed Board as well as key agencies of the federal 
government is to begin work early on developing a consensus strategy that will, at a minimum, address the 
following topics for the total education and training community and the relevant policy makers responsible 
for those institutions: 



► Develop an information dissemination strategy to explain the intent and potential; 

► Develop and assist the implementation of technical assistance to help promulgate the 
utilization of the standards; 

► Develop and assist organizations that would have responsibility for being the "linking 
institutions" between the standard setting and assessment bodies and the education and 
training provider community; 

► Address the issue of staff development, and y 

► Develop improved quality assurance mechanisms. 



Develop an aggressive information dissemination agenda. 

At this point in the history of skill standards, one of the tasks is to help people understand what is 
meant by skill standards and how they can help improve the education and skills preparation of individuals. 
While the proposed Board should be involved in this effort, in no way should this be a "solo" effort. Several 
parts of the U.S. Departments of Education and Labor need to be involved. 
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Teachers, counselors, principals, superintendents, local school board members, community college 
instructors, state departmental staffs responsible for approving programs within community colleges or 
developing curriculum frameworks for the public schools and, chief state school officers will need to 
understand the implications of skill standards. So do representatives of employment and training 
organizations such as PIC's and community based organizations. For many of these audiences, tapping 
into their national peer networks for purposes of sharing information should be a beginning step. There will 
be a need to ensure that each of these different audiences are receiving the same core information 
regarding the skill standards in order to reduce confusion and/or promotion of "turf fights" that have all too 
often characterized the relationships between the various parts of the education system and among 
education and training providers. 

Invest in technical assistance and support. 

Consortium-type efforts involving one or more national networks may be a cost-effective approach 
to develop materials related to the use of the standards. For example, it may be desirable to develop a 
project that would include representatives of state departments of education, community colleges and 
vocational technical education to provide technical assistance to their member institutions regarding how to 
develop articulation agreements between secondary and postsecondary institutions. They could use the 
identified skill standards in broad occupational areas and build upon the Tech-Prep tiered approach to 
developing curriculum. Consortium networks, such as the recently established Alliance for Curriculum 
Reform, representing 26 national subject-matter professional organizations and other education groups 
involved in curriculum development based upon national standards could be tapped to help think through an 
infusion strategy. 

A lesson could be taken from Japan regarding the use of association networks in the skill 
standards arena. One of the primary activities of the Vocational School Principals Association is to 
develop, in concert with industry research teams, curriculum and related tests that are then sold to the 
schools throughout the nation. Through this approach national curriculum is constantly updated and 
criterion-referenced tests are available, and trusted, by ail the relevant stakeholders. 

These suggestions are illustrative of possible approaches that could be taken to help organizations 
and institutions understand the value and utility of skill standards. Thinking broadly in terms of reaching 
out, not through the all too typical narrow silos of categorical funding is essential. 

Adding focus and emphasis to currently existing research and technical assistance efforts should 
be aggressively pursued. A careful review of how to infuse skill standards related activities into existing 
efforts would seem appropriate, across the full spectrum of federally supported research and development 
activities (e.g. OERI funded research centers and regional laboratories, OVAE's research centers for 
vocational education and literacy, the U.S. Department of Commerce support for advanced technology 
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centers, etc..) With some common strategic planning several of these on-going efforts could contribute 
value-added support to the effort. 

Develop linking institutions. 

These suggestions could prove useful but still insufficient to ensure the goal of improving 
education and training systems primarily due to the temporary nature of the suggested tasks and the 
presumption that the funding would come from short term federal government grants or contract funds. 
Institutions that are sustainable, and jointly "owned", we believe will be necessary. For lack of a better 
descriptor, we have called these, linking institutions. There are several approaches that could be 
considered for the development of such institutions. For example, organizations could be established 
around major occupational clusters or geographic regions. The essential point is that industry 
representatives, state governments, and especially the representatives of secondary and post-secondary 
institutions must come together and agree upon a common agenda that will continuously translate skill 
standards into curriculum, update curriculum, instructional materials, and make it widely available to all 
types of education and training institutions. 

Substantial attention must be given to developing the most cost-effective means to ensure that 
students have access to the appropriate instructional materials. Many individual schools have found that 
fiscal constraints do not permit them to provide the necessary up-to-date equipment needed for specific 
occupational training. Some creative thinking is needed in this arena. It may be desirable to think about 
creating "teaching factories" where the government and the private sector enter into agreements for the 
purpose of teaching students, or more aggressively using the apprenticeship model for education. Again, 
the suggestions are made, not to promote a single prescription but to counsel that new and different 
creative solutions will be essential. 

One must be impressed, we believe, with the experience of voluntary multi-state collaborative 
efforts, such as V-Tecs. It shows the value — indeed the necessity — of having organizations that are 
translators of skill standards into curriculum. It has shown the value of sharing development costs across 
states. 

Develop staff capacity throughout the education and training networks. 

This issue was consistently identified as a concern. Instructors from all types of institutions need 
to have a deep understanding of what is happening inside workplaces. Finding qualified instructors will 
continue to be one of the most important barriers to ensuring quality education and training. 

The study did not generate possible solutions within this arena but simply highlighted how 
essential the issue is in almost ail education and training situations. Again, it would be inappropriate for the 
proposed Skill Standards Board to be expected to solve this problem. Attention must be given to this issue 
by several parts of the U.S. Department of Education, the states and several different organizations. An 
observation may be helpful. There has been, over time, a substantial amount of federal funds used to 
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support staff development but much of the support has been piecemeal in nature. It would be best to 
develop a holistic strategy, in concert with the states and key organizations representing teachers and 
other personnel. 

Develop improved quality assurance mechanisms for education and training programs. 

Other countries not only set competency-based outcome standards but also quality standards for 
programs, regardless of the institutional setting. Germany, for example certifies learning places that are 
authorized to receive some type of funding or recognition in the system. 

The accreditation community and state governments need to be substantially engaged in 
consultations that focus on how to develop improved quality assurance processes for recognizing programs 
of quality. 

Another component of improving quality assurance mechanisms can be found within performance 
management systems promulgated by the federal and state governments. Both levels of government 
should commit to incorporating skill standards within all of their performance management systems for 
workforce development programs. 

QUESTION 5: WHAT TESTING GUIDELINES SHOULD BE RECOGNIZED TO INSURE AGAINST 
DISCRIMINATION BASED ON GENDER, ETHNICITY OR DISABILITY STATUS? 

Build upon the guidelines that have already been established by private, voluntary national standard setting 
quality assurance organizations. 

There are two organizations that have developed strong and well researched testing guidelines for 
use in standard setting and assessment programs; the American National Standards Institute (ANSI) is the 
U.S. member body of the International Organization for Standards (ISO) and the National Commission for 
Certifying Agencies (NCCA). The guidelines vary only in small degrees, probably reflecting the history and 
traditions of the organizations themselves. Both organizations' guidelines, if followed, would assist any 
testing efforts to ensure against discrimination based on gender, ethnicity or disability status. 

A substantial amount of attention needs to be given to how the published rules are developed, 
how many languages the tests will be made available in, whether only citizens or any U.S. resident can take 
the tests, who pays for the assessment, and much more. But as noted, there is substantial experience that 
can be used to build satisfactory guidelines. 

QUESTION 6: TO WHAT EXTENT DO THE CURRENT APPRENTICESHIP AND COOPERATIVE EDUCATION 
SYSTEMS IN THE U.S. PROVIDE A FRAMEWORK FOR SETTING SKILL STANDARDS? 

Neither the registered apprenticeship system nor the cooperative education system, in and of 
themselves represent a framework for setting and implementing skill standards. However, both systems 
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provide lessons for the new system from their implementation history, and both may be regarded as 
potential points of articulation between systems currently administered by the Departments of Labor and 
Education and the new skill standards system. 

The vocational-technical education system, within which many of the co-op programs are 
embedded, provides a larger context within which to discuss the development of the framework for setting 
skill standards. 

The Registered Apprenticeship Programs 

In general terms, the framework of registered apprenticeship programs has four elements in 
common with the envisioned framework of the new skill standards system: 



Apprenticeship is a federal-state system. Since not ail states have State Apprenticeship Councils 
(SACs), the state partnership with the federal Bureau of Apprenticeship and Training (BAT) is incomplete. 
Further, state autonomy has resulted in some inconsistency across states. Finally, the structure of the 
system does not assure portability across jurisdictions of credentials issued under this system. 

With respect to management-labor, apprenticeship programs require participation by employers in 
all sponsorship arrangements and, where the workforce is organized for collective bargaining, also require 
participation by labor (or an explicit waiver). Apprenticeship does not involve worker representation if the 
workforce is not organized. A majority of registered apprentices are enrolled in joint programs, and the 
joint programs include the largest programs and those generally regarded as highest in quality. Therefore, 
registered apprenticeship has a de facto association in the minds of many with organized labor. 

Apprenticeship involves an industry-education partnership because registered programs must 
provide related instruction. The providers include public secondary, public postsecondary and private 
institutions. The instructional component is distinctly secondary to the experiential component and not 
always well-articulated with it. 

With respect to the internal structures and procedures of registered apprenticeship programs, 
including their quality control provisions, variability is the rule rather than the exception. Traditionally, 
registered programs have been time based but competency-based programs are gaining in frequency. Ail 
registered programs emphasize informal observation of skills mastered and demonstrated on the job, 
rotation through various aspects of the trade, and progressive advancement in wages. Less standard is the 
degree to which skill requirements are updated and validated, the types of assessment employed (e.g., 
informal observation, formal written, formal performance), and the training and certification requirements of 
related instruction instructors. 



It is a federal-state system; 

It is a management-labor system; 

It is an industry-education system; and, 

It is a public-private system. 
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In the area of incentives and disincentives, the experience of registered apprenticeship offers some 
interesting lessons. With respect to incentives, the Davis-Bacon Act provides substantial financial 
incentives for registered apprenticeship in the construction trade and, in fact, the majority of registered 
apprentices are in construction occupations. With respect to disincentives, two factors appear to have 
been operative, particularly outside the construction industry. First, the equal opportunity provisions and 
procedures of registered apprenticeship programs have caused some employers to shy away from 
volunteering for a level of government oversight that they would not otherwise be subject to. Similarly, the 
de facto association between apprenticeship and organize 1 , labor has caused some determinedly non-union 
employers to fear that the adoption of apprenticeship wo-.'d be a stepping-stone toward an organized 
workforce. 

With respect to the public-private character of apprenticeship, it involves public sector 
"accreditation" of private industry entities that are training providers. It is within this area that 
consideration could be given by the National Board to establish specific criteria for the recognition of a 
pubic-private apprenticeship program that accommodates the strengths of the apprenticeship system but 
helps correct some of the identified weaknesses {eg. does not cover he entire career, assurance of 
continual updating of standards, improved assessment processes.) 

Australia, found that it was useful to establish criteria to recognize an individual company's 
standards when the company controlled a significant segment of an industry. There is no reason this 
country's system cannot be sufficiently flexible to accommodate the unique features of the apprenticeship 
programs. 

Canada offers an excellent example of how to create portable and easily recognized qualifications 
based upon the traditional form of apprenticeship training. The processes used to develop their Red Seal 
exams offer a tested approach for qualifying individuals within this highly decentralized, privately sponsored 
but publicly supported form of training. 

The federal Bureau of Apprenticeship Training's (BAT) regulations predominately focus on process 
and program structure issues and the only major "quality assurance" standard contained in those 
regulations relates to wage rates. The standards expressed in the registration document and included in 
the individual apprenticeship agreements that serve as contracts for employment are most often time 
based, not competency based. Most of the attention is focused on the initial preparation of workers, and 
little attention is given to the continual upgrading of the workforce and few stress the entire career path. 
Furthermore, the process standards do not require continual updating and validation of the skill standards, 
nor do they promote the use of the best practices in assessment. The emphasis placed on assuring quality 
instruction by many of the national BAT p ograms appears to be of vital importance in the conduct of a 
quality apprenticeship program, however, the BAT regulations are silent on the issue of instructor 
qualifications. 

Individual BAT-recognized apprenticeship programs do provide excellent examples of how to link 
industry skill requirements to related instruction and then certify individuals based on a nationally 
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recognized third party assessment set of instruments that have been constructed based on the criteria 
embedded in nationally available training materials. The study highlights three such examples, the Laborers 
Union's/AGC new assessment certification initiative, the International Masonry Institute's initiative, and the 
National Joint Apprenticeship and Training Committee of the International Brotherhood of Electrical 
Workers and the National Electrical Contractors Association. There are other apprenticeship programs of 
high quality throughout the country and there is little doubt that the essential features of the apprenticeship 
model for learning in the classroom as well as in the workplace represents, if not the best approach for 
assuring a trained workforce, is close to the top as a training model. 

In all, registered apprenticeship offers some interesting parallels for consideration by those involved 
in developing the new skill standards system. As identified above, there are a number of general 
similarities, as well as some key differences. Two limitations of registered apprenticeship appear to 
deserve particular consideration. First, registered apprenticeship is characterized by considerable variability 
and fragmentation, some of which may be necessary and desirable in a diverse and far-flung economy such 
as ours, and some of which may be excessive and counter-productive for a national system. Second, 
registered apprenticeship has achieved only very limited implementation, despite obvious virtues, and 
despite a variety of efforts to promote more widespread implementation. The factors associated with 
these two limitations deserve consideration by the National Skill Standards Board so that the positive 
aspects can be maximized and the negative aspects minimized. Their involvement in the development of 
the national voluntary skill standards initiative can only enhance the total effort and improve the 
Apprenticeship programs simultaneously. 

Cooperative Education Programs (Co-Op) 

Although many cooperative education programs utilize skill standards in their agreements between 
business, education and student, it is not one of the drivers in the standards development process. The 
cooperative education program is frequently used as the capstone to workforce related training, as a means 
for secondary or postsecondary students to practice skills learned in the classroom and in an internship 
laboratory setting. In other words, Co-Op programs are users of skill standards that have been incorporated 
into course syllabi and training agreements to determine the skills which will be taught or learned in the 
classroom and which at the job site. 

Evidence clearly shows that cooperative education or internships are considered integral parts of 
learning experiences. Several of the industry and professional associations who submitted materials 
regarding their involvement in specialized accreditation programs noted requirements for some form of 
applied learning off campus. The Title VIII of the Higher Education Act and the Carl D. Perkins Act both 
support campus based co-op programs in several forms. Certain occupations, such as engineering and the 
allied health professions (which typically use the term internship) have a long and rich history in such 
programs. However, these programs may or may not be rooted in national, state or local skill standards 
systems. 
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The division of what is learned in the classroom and on the job, may require additional research 
as the skill standards effort unfolds and improved curricula and instructional methods are developed. 
Cooperative education programs should be well positioned to assist in such research and development 
efforts. 

The Vocational-Technical Education System 

The "parent" authorizing legislation for the co-op education programs, particularly for those at the 
secondary level, has traditionally been the vocational-technical legislation. It is important to discuss how 
this system can support the development of a national skill standards system simply because it will have a 
broader impact than that of the co-op education program. 

It is probably a safe assumption that a substantial burden of implementing the educational 
components of the occupational preparation for the workforce will fall upon the secondary and 
postsecondary institutions that are as a whole called the voc-ed system. Clearly, the work within the 
states, to date, to establish skill standards and related competency based curriculum for occupationally 
related education has been accomplished primarily through the voc-ed structure. Also the challenge to 
improve the articulation between secondary and postsecondary education institutions has been spurred on 
by the establishment of the Technical-Preparation program within the Perkins II legislation. The effort to 
increase the integration of academic and vocation related curriculum is also being stimulated by the Perkins 
II legislation. 

The vocational technical education system's work with over 700 employer committees to establish 
skill standards for entry and "journeyman" levels of skills mastery makes this system a leader in this 
emerging field. While this provides an important base, for the most part, the number of occupations for 
which skill standards have been or are bsing developed has been driven more by the types of programs 
than by the prevalence of different occupations or the changing nature of work, though presumably this is 
reflected in the content of the programs offered. There are few nationally validated and commonly used 
(across all states) skill standards within the vocational education system and there is little evidence of 
consistency in the level of education and occupational specialization for which standards are being set. 



Nor are there yet many examples of portable "credentials" that accompany nationally industry validated 
standards. 

The study documented that the primary use of occupational skill standards has been to develop 
task lists as the basic units of curriculum, instruction and evaluation criteria for entry to sometimes 
"journeyman" levels. The study indicates that employers have been willing to work with the state officials 
on all three of these tasks. However, it also reveals the additional opportunities need to be pursued. 
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Have states collectively examine the current base of occupational skill standards. 

Collective work among all of the states, in concert with the National Skill Standards Board, could 
be undertaken to look across all major occupational areas for which standards have already been developed 
to ascertain where it would be possible to (a) identify common standards that could be used within a 
national framework using common language and competencies, and (b) assess where it would be possible 
to improve the balance between supply of identified occupational competencies (e.g. there were 120 
identified for agriculture and 62 for health). While numbers and lists do not tell the whole story they are an 
indicator of a level of effort that may not mesh with the highest needs of the labor market. 

Additionally, education institutions, more so than training institutions, have the responsibility to 
provide students with the theory and knowledge that will prepare students for the future. Therefore, while 
the skill standards provide an essential baseline of competencies the "related instruction" needs to help 
students be prepared for the evolving workplace. This is one of the critical reasons the "SCANS Skills" 
have high utility for the education community. They can be used to help develop the framework of a 
continuum of standards that will be needed in the future. 

Support an R&D effort to identify and measure the competencies that can best be learned within the 
workplace versus the classroom. 

Much of the work that has been done to date regarding skill standards within the education 
community has been focused on eliciting the support of industry to help identify the competencies required 
in the workplace that can be translated into curriculum for the classroom. However, more attention is 
needed to sort out which types of competencies are best learned at the work-site, under what conditions, 
versus what competencies are best learned in a classroom or a laboratory. This research should be linked 
to testing new forms of evaluations and assessments that generates a tighter link between the employers 
and classroom instructors — yet is linked to testing and developing "portable credentials" . 
Improve the pipeline for assuring high quality professional capacity 

Work among the institutions of higher learning, the vocational education research community, and 
the state policy makers is needed to help the vocational education community to continue to improve and 
expand competency-based curriculum and instructional techniques. Work is also needed to develop 
improved student assessment methods that could eventually be incorporated into a commonly recognized 
portable credentialing effort. There are most assuredly, a multitude of other issues regarding improving 
the pipeline of professionals emerging from the institutions of higher education but these specific 
suggestions focus on the implications emanating from a skill standards system. The Office of Vocational 
and Adult Education is well positioned to take the lead to encourage such efforts. 
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Mesh the use of Skill Standards with the Development of Performance Standards for Vocational Education 

The Perkins ii legislation requirement that states develop performance standards for secondary and 
post-secondary programs should, overtime, include the use of skill standards. In order for this integration 
to be achieved it will be necessary for the key consultants who are assisting the federal government and 
the states in development of performance standards to become knowledgeable and involved in the 
emerging skill standards initiative. 

A portion of this work should address how both the performance standards and the skill standards 
can be used to assist state and local operators use both forms of standards to continuously improve both 
program quality and linkages with the employing community. 

ADDITIONAL OBSERVATIONS 

There are additional observations that were not explicitly required to be addressed in the contract, 
but emerged in the study that merit consideration by the federal government. 

Use the common language of standards in classification systems. 

It will be important to pay attention to the issue of using common language across several 
standard information systems such as the classification of occupations and industries, (i.e. the Dictionary of 
Occupational Titles, the Standard Occupational Classifications and the Standard Industrial Classification, 
etc.) 

Canada has found several useful ways to help facilitate the occupational counseling and labor exchange 
services by coordinating the development cf common classification systems across several programmatic 
areas. 

Inform the general public and support enhanced counseling services. 

A substantial effort must be made to help individual students and their parents iearn about the skill 
standards and their implications, individual counselors responsible for helping students in a wide variety of 
institutions should have a deep understanding of the career and educational options that will be derived 
from meeting the standards. 

The Job Service and career guidance and counseling staffs housed in schools and training 
institutions should have easy access to information about the standards as well as which local 
organizations provide what type of skill standards related training. 

All career information systems should incorporate information on skill standards and credentials. 
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CONCLUSIONS 

The effort will very much be a part of a reinventing government agenda. It will require not just 
reaching a consensus on the how to develop a common framework within which to organize and develop 
skill standards but the development of the capacity within both the public and private sectors to support 
individuals' education and training to those standards. 

The IEL team became convinced that the development of the standards, while time-consuming, will 
prove to be one of the easier tasks to fulfill. There is a rich body of experience in this country about the 
identification of skill requirements and the development of validated common standards that have been 
aided by civil rights laws and regulations geared towards assuring non-discrimination. The critical next step 
is to bring together the skill standards and credentials resulting from these experiences under a common 
framework that results in nationally recognized standards and portable credentials. 

A large challenge, the team believes, will be the development of the capacity to exploit the 
standards to their fullest potential within organizations and governments. This will require developing 
consensus and common action plans among various parts of the federal government as well as within 
industry networks and sub-national units of governments. This effort cannot be accomplished only by the 
establishment of the National Skill Standards Board, but will require concerted and coordinated action on 
the part of several federal Departments. This will require new approaches to interagency collaboration. 

The reality is that all of the processes, including the development of a common framework within 
which the skill standards are established, within and across industries, will be a politically negotiated 
process among the various stakeholders and will take several years to develop. This means that the 
structures and systems that will be established need to be sufficiently flexible so that continuous 
refinements and improvements will be encouraged. 

It is essential that a wide array of capacity building efforts be established that focus on the needs 
of education and training organizations if the skill standards initiative is to become a part of the human 
resource development system in the United States. Other countries, including those that have had 
traditionally strong federal government presence in setting education policy, have found it necessary to 
develop special support systems for the education and training providers to establish the appropriate 
linkages with the industries within this arena. The non-directive role that both the federal (and arguably the 
state governments) have over education and training institutions in this country makes it essential that 
substantial attention and careful thought be given to developing sustainable institutional arrangements. 
Agreement will need to be reached among the levels of governments responsible for supporting education 
and training systems regarding the best approaches to be pursued. 

A quiet and deliberate approach that is also aggressive will be required to generate the mutual trust 
among the private and public sectors, among the potential private sector partners, among the different 
levels of government, among different types of education and training institutions, between different 
agencies within the same federal departments and across federal departments. The IEL team came to 
believe the value of establishing skill standards was worth the challenge. 
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APPENDIX A 

LIST OF ORGANIZATIONS UNDER CONTRACT 
WITH FEDERAL GOVERNMENT TO DEVELOP 
SKILL STANDARDS 
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DEPARTMENT OF EDUCATION AWARDS (Round 1) 







Foundation for Industrial Modernization 
Washington, DC 


Computer-aided drafting 
(cluster of occupations) 


Education Development Center 
Newton, Massachusetts 


Biotechnical sciences 
(laboratory technicians) 


Far West Lab for Educational Research and 

Development 

San Francisco, California 


Health science and technology 
(occupational clusters) 


National Automotive Technicians Education 

Foundation 

Herndon, Virginia 


Automotive, auto body and truck 
(technicians) 


Graphic Arts Technical Foundation 
Pittsburgh, Pennsylvania 


Printing (occupational clusters) 


Electronic Industries Foundation 
Washington, DC 


Electronics (technicians) 


Vocational Technical Education Consortium 
of States (V-TECS) 
Decatur, Georgia 


Air conditioning, refrigeration and power 


DEPARTMENT OF EDUCATION AWARDS (Round 2) 






Foundation for Industrial Modernization 
Washington, DC 


Advanced Manufacturing 


Human Services Research Institute 
Cambridge, Massachusetts 


Human Services Occupations 
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DEPARTMENT OF EDUCATION AWARDS (Round 2 Continued) 



Laborers-AGC Education and Training 
Fund 

Pomfret Center, CT 


Health science and technology 
(occupational clusters) 


American Chemical Society 
Washington, DC 


Chemical Process Industries 


Center for Occupational Research & 

Development 

Waco, Texas 


Hazardous Materials Management 
Technician 


Center for Occupational Research & 

Development 

Waco Texas 


Photonics Technician 


American Welding Society 
Miami, Florida 


Welding 


Grocers Research and Education Fund 
Sterling, Virginia 


Food Marketing Industry 


National FFA Foundation 
Madison, Wisconsin 


Agriscience/Biotechnology 


US DEPARTMENT OF LABOR AWARDS 


Institute of Industrial Launderers 
Washington, DC 


Industrial Launderers 


Council on Hotel, Restaurant and 
Institutional Education 
Washington, DC 


Tourism, Travel and Hospitality 


National Tooling and Machining 
Association 

Ft. Washington, Maryland 


Metalworking 


American Electronics Association 
Santa Clara, California 


Electronics 


National Electrical Contractors Association 
Bethesda, Maryland 


Electrical Construction 


National Retail Federation 
Washington, DC 


Retail Trade 
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The Institute for EducationaJ'teadership 

Center for Workforce Development 
1001 Connecticut Ave., NW, Suite 310 
Washington, DC 20036 
Telephone (202) 822-8405 
FAX (202) 872-4050 
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